-
Notifications
You must be signed in to change notification settings - Fork 112
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Reverseindex search #369
base: master
Are you sure you want to change the base?
Reverseindex search #369
Conversation
hm, I'm scared of the performance :( |
yeah I agree... |
that's a good idea, can even just only update after they leave a row, maybe? and you can also debounce the rest of the updates (it's hard to guarantee the index is 100% correct, right? i'd be okay with small race conditions that get fixed when you revisit the row) |
I changed it to update after leaving the row. None of the updates are blocking, so there might be some race conditions leading to an inconsistent index. The search function is robust to false positives since it manually checks it again. All new rows in the last session are updated again in the next session so there shouldnt be false negatives. Also since timing isnt as critical now it indexes all prefixes of words. The main performance bottleneck now is getting the search results text for display which seems unavoidable? Maybe debounce the search function itself or only search on enter? |
I'm having a lot of trouble debouncing search. Do you see how it could be done? Also still have no idea why rendering results takes so long :/ Would appreciate any help if you have time! |
i think you want to debounce this line in menu.ts
it looks like it might be straightforward, but i wouldn't be surprised if it ended up very tricky regarding the rendering results, you can profile in chrome i think. is it worse on your branch than on master? |
I think this pr is ready to be reviewed, I'll leave debounce for the future. In its current state this pr makes search very usable for large documents, just typing search queries is a bit sluggish. When testing I recommend making a backup of your data, it will add a lot of stuff to your database. It also takes a while to fully initialize, minutes for me. |
Makes search fast for large documents.
The main constraint is database access time with firebase. On average it takes 3 database queries to maintain the reverse index every time a row is changed, which is pretty much the limit before typing feels sluggish.
To get the number of queries down, there are compromises. You can only search for whole words, you must type out the entire word. Only 20,000 rows are stored for any word, so the search will fail in edge cases where every search query is super common.
todos:
show initialization progress
rewrite search tests