Commits

• 1c3c309: Cumulative updates (#93) (MilesQLi) #93

Cumulative updates.

Release Note

• UI updates for server and IDA Pro plug-in
• Improve Documentation
• Allow to reset parameters in workbench
• Allow to append files multiple times to index and search

SHA-256:

Kam1n0-IDA-Plugin.tar.gz:
160f2838cba66d0e1fdf579d2e916923295e20a4cc42c51ef172e119b2f35ece

Kam1n0-Server.tar.gz:
37915fa4172e3b6690e795f62c4ec4a4bc67477ba229ecd807d3531e581a1a6e

Kam1n0-IDA-Plugin.exe:
f608fc5fad1f73d7cb4b91f21835d42fb2dbece79ab039c6a6d599325b8f0ba8

Kam1n0-Server.exe:
910c3b405d581c450811eed34e4845d92123de2808503f67f08e5747577f806e

Breaking changes. Much faster clone search and indexing (1M functions less than 40 minutes). Smaller memory footprint. Fixed several critical bugs in Asm-Clone.

Release Note

Platform:
1/ Functional - fixed the binary/function/block counter for each application
2/ Functional - optimized Cassandra in query using binning
3/ Functional - optimized Spark job pool scheduler setting to lower the timeout error and the OOM error
4/ Functional - fixed the read-after-release error in the Spark job monitor
5/ Functional - Stored additional meta data

Binary Composition Analysis:
1/ Functional - able to filter out queries by the number of basic blocks
2/ Functional - added a new running job procedure to dump clone metadata as a JSON file
3/ Functional - fixed the address range display order (sorted)
4/ Functional - fixed the atomic update issue of the clone result summary
5/ UI - option to dump result as JSON file (on upper-right corner)
6/ UI - show progress and estimated completion time
7/ UI - the summary box in composition analysis shows
[matched target functions]/[number of target functions]/[number of source functions]

Asm-Clone:
1/ UI - show [function name - # basic block] at the query page
2/ Query optimization - a recursive bucket split procedure for better efficiency
3/ Query optimization - two level filtering to reduce the memory footprint
4/ Query optimization - lazy loading of the source block information to reduce the memory footprint
5/ Query optimization - local mode Spark optimization for better searching efficiency
6/ Index optimization - local mode Spark optimization for better indexing efficiency
7/ Index optimization - Cassandra query optimization to reduce timeout errors and large frame errors
(to fix the missing blocks and functions issue)
8/ Index optimization - increased the default read timeout setting
9/ Index correction - fixed hashing prefix starting length issue
(to fix the missing blocks and functions issue)

SHA-256:

Kam1n0-IDA-Plugin.tar.gz:
ea61a454e29b7b7aa54951a03d7e717f936e1997b6f3f25651e8a47118c538d4

Kam1n0-Server.tar.gz:
208629eb54311406985839729d396a6361ede1da80f3c8a90cdf1a748cfb1616

Kam1n0-IDA-Plugin.exe:
f3da953cd47e66a3769fbb683eaf1783f3c22282611afb99afc38a087d179b9e

Kam1n0-Server.exe:
00779d5f93a40a99b435d2e3f98fe25a3e51ecad1797614c3707f07d36f0893a

Kam1n0 v2:

Latest tag for 2.0.0:

PullRequest-169_2018-05-23-21_*

• Breaking changes, cumulative updates.
• Added several views for Sym1n0 (vex text/flow comparison, syntax tree comparison).
• Sym1n0 is not backward compatible.
• First RC release.

Checksum:

Name: PullRequest-169_2018-05-23-21-33-53_Kam1n0-IDA-Plugin.tar.gz
SHA256: 0842CC3F4CC4F268A91B34A80E2E42C76456E4E598583430C12200FF1BB189E2

Name: PullRequest-169_2018-05-23-21-33-05_Kam1n0-IDA-Plugin.exe
SHA256: A5D5B8721D3E56EDC6740A30FFEB4E382F3C4ED5737C51A7E3F26851D05A6A76

Name: PullRequest-169_2018-05-23-21-33-25_Kam1n0-Server.exe
SHA256: FE42D54D3606C0C2CD1A4C43F1CA31520E1E8A566B54510C60AFCFE7B1EE5895

Name: PullRequest-169_2018-05-23-21-33-54_Kam1n0-Server.tar.gz
SHA256: EF2B4CBB4B48E58D97524D3032BC6E9A43BFF22A9B0302F9152BD31622305665

Notes:

• Completely remove the old version first.
• Support any IDA > 6.7
• Backward incompatible.
• Refactored engine to support serving multiple repositories simultaneously.
  • User can create applications of different types.
  • An application comes with a repository and other resource.
  • A specific type of application provides a specific set of assembly analysis services.
  • Current application types:
    • Asm-Clone (sub-graph & architecture-agnostic).
    • Sym1n0 (cross-architecture search).
    • Asm2Vec (search whatever you index, robust, fast but no-subgraph).
    • More analysis will be added later.
• Redesigned and enhanced UI views and elements.
• Added assembly code representation learning (against optimization and code obfuscation).
• More details can be found on the user manual (coming soon).
• Linux support in progress.
  • *.tar.gz files are for linux.
  • Preliminary tested on WLS (Ubuntu distribution)
  • Dependencies for IDA Plug-in are not included. Check Github page for required dependencies.

Tag history:

Issue-166_2018-05-14-12-34-33_* (cumulative update)
PullRequest-157_2018-03-26-20-28-29_Kam1n0-* (initial v2 release)

Minor updates:

• Fix _idaapi module not found issue in IDA 6.9.5
• Added linux supports (with the help from @zaddach)

• [Kam1n0 Core] Added a new symbolic mode. Now it supports cross-architecture sub-graph clone search on the symbolic expression level. Included libvex and z3 library. Supported architectures: x86, AMD64, MIPS32, MIPS64, PowerPC32, PowerPC64, ARM32, and ARM64.
• [Kam1n0 Core] Updated graph search algorithm. Improved scalability & accuracy. Updated default ALSH settings.
• [Kam1n0 Core] Added Visual C++ Redistributable for VS15 dependency (included in the installer, it is for z3).
• [Web UI] In the symbolic mode, we also visualize the control flow graph with abstract syntax tree for each basic block.
• [Web UI] User can index multiple files at a time.
• [Web UI] User can directly index idb or i64 file.
• [Web UI] Fix web UI bugs and improve usability.
• [Web UI] User can interrupt running jobs through the administration portal.
• [RESTful API] The old API is no longer working. Check out new API after installation.
• [IDA Pro plug-in for Kam1n0] Support composition analysis query.

Release note:

• [Web UI] Added a web interface for clone search with an assembly function.
• [Web UI] Added a web interface for clone search with a binary file.
• [Kam1n0 Workbench] Added Kam1n0 Workbench for creating and managing multiple repositories on a single workstation.
• [Kam1n0 Core] The binary file clone search result can be shared and browsed on the other machine without access to the repository.
• [Kam1n0 Core] Support indexing and searching for large binary file (>40mb) without limits on system memory.
• [Kam1n0 Core] Support ARM, PowerPC, x86 and amd86 binaries.
• [Kam1n0 Core] Support user-defined processor architecture.
• [Kam1n0 Core] Optimized index structure supports better scalability and clone search quality.
• [Kam1n0 Core] Kam1n0 no longer skips basic blocks which have less than three lines of instruction. Now only single line basic block is skipped; thanks to the new index structure.
• [IDA Pro plug-in for Kam1n0] [Experimental] Added assembly fragment search functionality.
• [IDA Pro plug-in for Kam1n0] Added a tree view for browsing large number of clones.

Kam1n0 Engine with IDA Pro Plug-in