[go: up one dir, main page]
Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

cargo test --all-targets also runs binaries marked as test=false #8338

Open
Ekleog opened this issue Jun 6, 2020 · 1 comment
Open

cargo test --all-targets also runs binaries marked as test=false #8338

Ekleog opened this issue Jun 6, 2020 · 1 comment
Labels
A-cargo-targets Area: selection and definition of targets (lib, bins, examples, tests, benches) C-bug Category: bug S-triage Status: This issue is waiting on initial triage.

Comments

@Ekleog
Copy link
Contributor
Ekleog commented Jun 6, 2020

Problem
Running cargo test --all-targets apparently runs binaries even when marked test=false. This is a problem in particular with cargo fuzz targets, that do some linker dark magic and start trying to fuzz during the test session, failing as it wasn't built for it.

Steps
First, provision the following files:

Cargo.toml:

[package]
name = "foobar"
version = "0.1.0"
authors = ["Léo Gaspard <leo@gaspard.io>"]
edition = "2018"

# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html

[dependencies]
libfuzzer-sys = "0.3.2"

[[bin]]
name = "example-bin"
path = "example-bin/main.rs"
test = false

src/lib.rs:

#[cfg(test)]
mod tests {
    #[test]
    fn it_works() {
        assert_eq!(2 + 2, 4);
    }
}

example-bin/main.rs:

#![no_main]

use libfuzzer_sys::fuzz_target;

fuzz_target!(|_foo: &[u8]| {});

Then, running cargo test --all-targets should result in a message like the following:

   Compiling foobar v0.1.0 (/tmp/foobar)
    Finished test [unoptimized + debuginfo] target(s) in 0.64s
     Running target/debug/deps/foobar-a99608cf2788c81d

running 1 test
test tests::it_works ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

     Running target/debug/deps/example_bin-045b781c1c0562c0
WARNING: Failed to find function "__sanitizer_acquire_crash_state".
WARNING: Failed to find function "__sanitizer_print_stack_trace".
WARNING: Failed to find function "__sanitizer_set_death_callback".
INFO: Seed: 2622528302
INFO: -max_len is not provided; libFuzzer will not generate inputs larger than 4096 bytes
INFO: A corpus is not provided, starting from an empty corpus
#2	INITED exec/s: 0 rss: 24Mb
ERROR: no interesting inputs were found. Is the code instrumented for coverage? Exiting.
error: test failed, to rerun pass '--bin example-bin'

Possible Solution(s)
Maybe it'd be possible to just not call binaries that are tagged test=false, even when ran with --all-targets, seeing as test=false should clearly stipulate that there are no tests to be found here?

Notes

Output of cargo version:
cargo 1.41.0

Anyway, thank you for all you do on cargo! It's a great piece of software, and I just wanted to report this small hiccup :)
@Ekleog Ekleog added the C-bug Category: bug label Jun 6, 2020
@ehuss ehuss added the A-cargo-targets Area: selection and definition of targets (lib, bins, examples, tests, benches) label Jun 6, 2020
@kpreid
Copy link
Contributor
kpreid commented Aug 2, 2021 •
edited
Loading

I just ran into essentially this problem with my own fuzz targets, but not with --all-targets:

cargo test foo (test filtering) seems to also ignore test = false (even though the otherwise-equivalent cargo test -- foo doesn't). This seems to contradict the documentation, which says that (emphasis mine):

Setting targets to test = false will stop them from being tested by default. Target selection options that take a target by name ignore the test flag and will always test the given target.

--bins also has this behavior, even though one might expect it to narrow the set of targets, not widen them.

kpreid added a commit to kpreid/all-is-cubes that referenced this issue Aug 4, 2021
@kpreid
Take all-is-cubes/fuzz out of the workspace.
d9a1a37 
This works around a Cargo bug or infelicity
  rust-lang/cargo#8338
where `[[bin]] test = false` is sometimes ignored.
ecton added a commit to khonsulabs/nebari that referenced this issue May 2, 2022
@ecton
Switching code coverage args
635a348 
--all-targets activates bins, even if they have test = false

Refs rust-lang/cargo#8338
@crepererum crepererum mentioned this issue Aug 29, 2022
Open
sharonminer052 added a commit to sharonminer052/beanch-nebari-develop that referenced this issue Sep 14, 2022
@sharonminer052
Switching code coverage args
acbed3f 
--all-targets activates bins, even if they have test = false

Refs rust-lang/cargo#8338
@epage epage added the S-triage Status: This issue is waiting on initial triage. label Nov 20, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
A-cargo-targets Area: selection and definition of targets (lib, bins, examples, tests, benches) C-bug Category: bug S-triage Status: This issue is waiting on initial triage.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@ehuss @epage @Ekleog @kpreid