[go: up one dir, main page]

Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reverseindex search #369

Open
wants to merge 13 commits into
base: master
Choose a base branch
from
Open

Conversation

platers
Copy link
Contributor
@platers platers commented Feb 17, 2021

Makes search fast for large documents.

The main constraint is database access time with firebase. On average it takes 3 database queries to maintain the reverse index every time a row is changed, which is pretty much the limit before typing feels sluggish.

To get the number of queries down, there are compromises. You can only search for whole words, you must type out the entire word. Only 20,000 rows are stored for any word, so the search will fail in edge cases where every search query is super common.

todos:
show initialization progress
rewrite search tests

@WuTheFWasThat
Copy link
Owner

hm, I'm scared of the performance :(

@platers
Copy link
Contributor Author
platers commented Feb 21, 2021

yeah I agree...
do you think updating the datastore only after the user stops typing would work?

@WuTheFWasThat
Copy link
Owner

that's a good idea, can even just only update after they leave a row, maybe? and you can also debounce the rest of the updates

(it's hard to guarantee the index is 100% correct, right? i'd be okay with small race conditions that get fixed when you revisit the row)

@platers
Copy link
Contributor Author
platers commented Feb 25, 2021

I changed it to update after leaving the row. None of the updates are blocking, so there might be some race conditions leading to an inconsistent index. The search function is robust to false positives since it manually checks it again. All new rows in the last session are updated again in the next session so there shouldnt be false negatives.

Also since timing isnt as critical now it indexes all prefixes of words. The main performance bottleneck now is getting the search results text for display which seems unavoidable? Maybe debounce the search function itself or only search on enter?

@platers
Copy link
Contributor Author
platers commented Mar 30, 2021

I'm having a lot of trouble debouncing search. Do you see how it could be done? Also still have no idea why rendering results takes so long :/ Would appreciate any help if you have time!

@WuTheFWasThat
Copy link
Owner
WuTheFWasThat commented Mar 31, 2021

i think you want to debounce this line in menu.ts

  this.results = await this.searchFn(query);

it looks like it might be straightforward, but i wouldn't be surprised if it ended up very tricky

regarding the rendering results, you can profile in chrome i think. is it worse on your branch than on master?

@platers
Copy link
Contributor Author
platers commented Apr 5, 2021

I think this pr is ready to be reviewed, I'll leave debounce for the future. In its current state this pr makes search very usable for large documents, just typing search queries is a bit sluggish.

When testing I recommend making a backup of your data, it will add a lot of stuff to your database. It also takes a while to fully initialize, minutes for me.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants