If you're really curious and want a sure way to test whether something is an object or a modifier, you can use something called the 'did so' test. (Linguists create these kinds of tests all the time, so that they can uncover all the hidden structure in language that people just know subconsciously.)
Setting up the test
Let's test the prepositional phrases in this sentence as an example:
San went to the duck pond in the morning.
First, we need an anchoring sentence, which should include all the same basic components as our test sentence:
Mai went to the fair in the afternoon. San went to the duck pond in the morning.
We're going to replace parts of the test sentence with "did so", and see what sounds okay and what sounds weird.
Note: You might need to grab some friends for this, because everyone speaks a little differently and might disagree on what "sounds weird"—it's not that anybody is wrong; they just have a slightly different dialect with slightly different rules.
(Our assessment of "weirdness" at Writelike is based on the Standard Australian English dialect, but this example should also work for Standard British and Standard American English, and we think it's likely to work for other English dialects too.)
Applying the test
Here is how we would use 'did so' to test whether the phrase "in the morning" is an object or a modifier:
Mai went to the fair in the afternoon. San did so in the morning.
We've replaced the verb and the prepositional phrase that we are not testing with 'did so'. It sounds fine, which means that the times the subjects go ("in the afternoon" & "in the morning") are modifiers.
What about "to the duck pond"?
Mai went to the fair in the afternoon. San did so to the duck pond.
This should sound at least a little bit odd. This means that the places being gone to ("to the fair" & "to the duck pond") are objects.
Mai went to the fair in the afternoon. San went to the duck pond in the morning.
Want to see something a bit weird?
Let's change the verb to 'run':
Mai ran to the fair in the afternoon. San did so to the duck pond.
The second sentence stayed the same as when we used the verb 'to go', but now it sounds fine! That's because the verb 'to run' doesn't have any objects, so the prepositional phrase "to the duck pond" is now a modifier!
Mai ran to the fair in the afternoon. San ran to the duck pond in the morning.
Tricky, right? But you can see how running little experiments like this can help reveal the underlying rules of a language.