In the usability tests that we ran in conjunction with the control tests for the reply feature, we asked participants "How do you expect to know when this person responds to you?"

All of the participants who answered the question stated that they expected to receive some kind of notification. Most participants expected to get an email as well as an alert on the site in close proximity to their user name.
One participant said that they thought of the "Watchlist" as a separate location for viewing all of their replies.

Here are the summaries logging their feedback:

UT-A Expects to get an email and also expects to hear a sound or see some sort of notification alert on the website.
UT-B Assumes email notification as well as a visual notification next to her username
UT-C Expects to get an email notification or notification if they are on the site.
UT-E Assumes that there will be a nested reply underneath their response. Doesn't know how to get notifications to appear, thinks possibly next to their username or if they follow a watchlist.
UT-F Assumes the watchlist and the talk page. tabs will display notifications.
UT-G Assumes checking watchlist would send email or app notification
UT-H I think it will email me a notification because I clicked the watchpage.
UT-I Expects to get a notification email and a notification on the user page.

I still need to cross reference the actual videos saved on usertesting.com to pull quotes.

Quotes:

"I would expect that I could (in the options) enable an email saying you've got a reply and I'd click a link to quickly come here. Also there are some options where if I'm on the site here or somewhere else, I'd hear a sound or see a number with a notification, maybe it could pop in somewhere in the corner of the screen and show the message. Basically, I would expect to have it be clear on the website and also if im not actively on the website." ReplyV2.0A

"I clicked on the button that asks for updates, so I would think that I would get a notification either within this wiki or if its attached to an email, I'd assume that I'd get an email" Reply2AB.V.4.5 (the button here is the Watch This Page button)

"I don't know, I honestly don't know because I don't see any notifications.[Audible sigh] I guess this [points to watchlist] I seriously don't know. [Audible sigh] I honestly don't know how I'd get a notification for this. There's just nothing. There's nothing" -Replyv2ABV3.3 (everything in brackets here is a note from me.)

Re: the test logs: I just discovered that we actually asked the question on multiple tests, so the test log in the comment above is just one response. That said, I read through all of them (32 tests) and the participant assumptions are consistent in terms of notification expectations.