Avocado Workshop

Get Started. It's Free
or sign up with your email address
Avocado Workshop by Mind Map: Avocado Workshop

1. About me

1.1. Lukáš Doktor

1.2. Happy Software Engineer at Red Hat

1.2.1. RHTS (bash)

1.2.2. Autotest/virt-test (python, virtualization)

1.2.3. Avocado (~3 months)

1.3. _

2. Introduction

2.1. Avocado is a next generation testing framework inspired by Autotest and modern development tools such as git. Whether you want to automate the test plan made by your development team, do continuous integration, or develop quick tests for the feature you're working on, avocado delivers the tools to help you out.

2.2. What is Avocado for me

2.2.1. .

2.2.2. Program which stores results of anything I want

2.2.3. Binary which provides means to help me execute anything using various environments

2.2.4. Set of python* libraries to simplify writing tests

2.2.5. Something easily integrateable with Jenkins (or othe CI)

2.2.6. Tool which allows me to share the results

2.2.7. Most importantly tool which allows me to do all the above in all possible variants

2.2.8. .

2.3. Avocado structure

3. Topics

3.1. Basic usage

3.1.1. Execute binary ./scripts/avocado run /bin/true when installed use only "avocado run /bin/true" ./scripts/avocado run /bin/true /bin/false ./scripts/avocado run passtest failtest warntest skiptest errortest sleeptest

3.1.2. Help and docs ./scripts/avocado -h ./scripts/avocado run -h ... cd docs && make html

3.1.3. Execute binary and compare outputs Used for tests which should generate the same stdout/stderr Possibility to combine with other plugins (run inside GDB,...) ./scripts/avocado run ./DEVCONF/Basic_usage/output_compare.sh ./scripts/avocado run ./DEVCONF/Basic_usage/output_compare.sh --output-check-record all ./scripts/avocado run ./DEVCONF/Basic_usage/output_compare.sh # modify the file ./scripts/avocado run ./DEVCONF/Basic_usage/output_compare.sh ./scripts/avocado run ./DEVCONF/Basic_usage/output_compare.sh --disable-output-check # exit -1 ./scripts/avocado run ./DEVCONF/Basic_usage/output_compare.sh ./scripts/avocado run ./DEVCONF/Basic_usage/output_compare.sh --disable-output-check

3.1.4. Execute binary and parse results using wrapper Used to run the same steps with various wrappers For example run complex testsuite with perf/strace/... (viz. Wrapper section) Alternatively use it to "translate" existing binaries to Avocado (here) DEVCONF/Basic_usage/testsuite.sh nasty testsuite which doesn't return exit_code ./scripts/avocado run DEVCONF/Basic_usage/testsuite.sh ./scripts/avocado run DEVCONF/Basic_usage/testsuite.sh --wrapper DEVCONF/Basic_usage/testsuite-wrapper.sh:*testsuite.sh # remove FAIL test ./scripts/avocado run DEVCONF/Basic_usage/testsuite.sh --wrapper DEVCONF/Basic_usage/testsuite-wrapper.sh:*testsuite.sh # remove WARN test ./scripts/avocado run DEVCONF/Basic_usage/testsuite.sh --wrapper DEVCONF/Basic_usage/testsuite-wrapper.sh:*testsuite.sh More details in Wrapper section

3.1.5. Multiplex using ENV ./scripts/avocado run examples/tests/env_variables.sh ./scripts/avocado run examples/tests/env_variables.sh -m examples/tests/env_variables.sh.data/env_variables.yaml ./scripts/avocado multiplex -c -d -t -h ;-) This is pretty close to QA, right? More details in Multiplex section

3.1.6. Grab sysinfo ./scripts/avocado run passtest --open-browser

3.2. Jenkins/CI

3.2.1. Jenkins? Jenkins

3.2.2. Quick&dirty setup

3.2.3. avocado simple

3.2.4. avocado parametrized

3.2.5. avocado lots of tests

3.2.6. avocado multi multi scenarios multi archs multi workers (OSs)

3.2.7. avocado git poll

3.3. Writing tests in Python

3.3.1. Can be executed as Inside Avocado ./scripts/avocado run examples/tests/passtest.py As python scripts examples/tests/passtest.py

3.3.2. Logging Possibility to set test as WARN self.log.warn("In the state of Denmark there was an odor of decay") See warntest test Log more important steps self.log.info("I successfully executed this important step with %s args" % args) self.log.debug("Executing iteration %s" % iteration) self.log.warn("You already know me") self.log.error("Everyone should check these messages on failure") stdout stderr

3.3.3. Whiteboard ./scripts/avocado run examples/tests/whiteboard.py cat ~/avocado/job-results/latest/test-results/examples/tests/whiteboard.py/whiteboard

3.3.4. Advanced usage of params See Multiplexer section

3.3.5. Advanced GDB See GDB section

3.3.6. Avocado-virt See Avocado-virt section

3.4. Test on remote machine

3.4.1. remote ./scripts/avocado run passtest --remote-hostname --remote-username root --remote-password 123456 Connects Copies tests directories Runs avocado inside machine with --json --archive Grab and store results Report results localy --remote-no-copy

3.4.2. vm run in libvirt machine ./scripts/avocado run examples/tests/passtest.py importerror.py corrupted.py pass.py passtest --vm-domain vm1 --vm-hostname --vm-username root --vm-password 123456 --vm-cleanup Checks if domain exists Start it if not already started (--vm-clenaup) Creates snapshot call remote test (--vm-cleanup) Restore snapshot --vm-cleanup --vm-no-copy

3.4.3. docker plugin first think what we require docker run -v $LOCAL_PATH:$CONT_PATH $IMAGE avocado run $tests grab the results we don't require any remote interaction, only simple execution plugin initialization 1) name, status 2) configuration hook 3) activation hook class RunDocker Prerequisites prepare docker cmd copy tests to shared location class DockerTestResult Actual execution 1) Execute the tests 2) Grab the results class DockerTestRunner

3.5. GDB

3.5.1. ./scripts/avocado run DEVCONF/GDB/doublefree*.py

3.5.2. ./scripts/avocado run --gdb-run-bin=doublefree: DEVCONF/GDB/doublefree.py backtrace ...

3.5.3. ./scripts/avocado run --gdb-run-bin=doublefree DEVCONF/GDB/doublefree2.py without ':' it stops at the beginning we can use `ddd` or other GDB compatible programs don't forget to resume the test (FIFO)

3.5.4. ./scripts/avocado run --gdb-run-bin=doublefree: DEVCONF/GDB/doublefree3.py --gdb-prerun-commands DEVCONF/GDB/doublefree3.py.data/gdb_pre --multiplex DEVCONF/GDB/doublefree3.py.data/iterations.yaml OK, we reached the "handle_exception" function we want to investigate n n jump +1 Works fine, hurray

3.6. Advanced GDB

3.6.1. vim DEVCONF/Advanced_GDB/modify_variable.py.data/doublefree.c

3.6.2. ./scripts/avocado run DEVCONF/Advanced_GDB/modify_variable.py --show-job-log

3.7. Wrappers

3.7.1. Valgrind ./scripts/avocado run DEVCONF/Wrappers/doublefree.py --wrapper examples/wrappers/valgrind.sh:*doublefree --open-browser cat /home/medic/avocado/job-results/latest/test-results/doublefree.py/valgrind.log.*

3.7.2. Strace The same way as valgrind Imagine for example crashing qemu machine running windows under very complex workload defined in test

3.7.3. Other ltrace perf strace time valgrind

3.7.4. Custom wrappers qemu on PPC wrap executed programs grab results

3.8. Multiplexer

3.8.1. Motivation QA Engineer walks into a bar. Orders a beer. Orders 0 beers. Orders 999999999 beers. Orders a lizard. Orders -1 beers. Orders a sfdeljknesv by_order:     a_beer:         msg: a beer     0_beers:         msg: 0 beers     999999999:         msg: 999999999 beers     lizard:         msg: lizard     negative:         msg: -1 beers     noise:         msg: sfdeljknesv

3.8.2. How would you solve our problem? !multiplex vs. !join virt: hw: cpu: intel: amd: arm: fmt: !join qcow: qcow2: qcow2v3: raw: os: !join linux: !join Fedora: 19: Gentoo: windows: 3.11: map tests to test->files

3.8.3. Usage ./scripts/avocado multiplex -h ./scripts/avocado multiplex -t ./scripts/avocado multiplex -t DEVCONF/Multiplexer/nomux.yaml ./scripts/avocado multiplex -t DEVCONF/Multiplexer/simple.yaml ./scripts/avocado multiplex -t DEVCONF/Multiplexer/advanced.yaml ./scripts/avocado multiplex -c ./scripts/avocado multiplex -c DEVCONF/Multiplexer/simple.yaml ./scripts/avocado multiplex -c DEVCONF/Multiplexer/advanced.yaml ./scripts/avocado multiplex -cd examples/mux-selftest-advanced.yaml eg. variant: 21 ./scripts/avocado run examples/tests/sleeptenmin.py -m DEVCONF/Multiplexer/simple.yaml ./scripts/avocado run examples/tests/sleeptenmin.py -m DEVCONF/Multiplexer/advanced.yaml ./scripts/avocado multiplex --filter-only/filter-out ./scripts/avocado multiplex -c DEVCONF/Multiplexer/advanced.yaml --filter-only /by_method/shell 2nd level filters Additional tags !using !include !remove_node !remove_value !join

3.8.4. Let's go crazy ./scripts/avocado multiplex DEVCONF/Multiplexer/crazy.yaml

3.8.5. Future Currently we update environment from used leaves by_something:     first:         foo: bar by_whatever:     something:         foo: baz Soon we want provide the whole layout by_something:     first:         hello: world         foo: bar by_whatever:     something:         foo: baz Automatic multiplexation 1) without multiplexation 2) test's file multiplexation 3) global file multiplexation

3.9. Eclipse

3.9.1. Debugging

3.9.2. Remote debugging download pydev copy it to $PYTHONPATH or add path in code:     import sys     sys.path.append('$PATH_TO_YOUR_PYDEV') import pydevd pydevd.settrace("$ECLIPSE_IP_ADDR", True, True) 1. True = forward stdout (optional) 2. True = forward stderr (optional)

3.10. Install Avocado

3.10.1. Using from sources # Uninstall previously installed versions git clone https://github.com/avocado-framework/avocado.git cd avocado ./scripts/avocado ...

3.10.2. git git clone https://github.com/avocado-framework/avocado.git # pip install -r requirements.txt python setup.py install avocado ...

3.10.3. rpm sudo curl http://copr.fedoraproject.org/coprs/lmr/Autotest/repo/fedora-20/lmr-Autotest-fedora-20.repo -o /etc/yum.repos.d/autotest.repo sudo yum update sudo yum install avocado avocado ...

3.10.4. deb echo "deb http://ppa.launchpad.net/lmr/avocado/ubuntu trusty main" >> /etc/apt/sources.list sudo apt-get update sudo apt-get install avocado avocado ...

3.11. avocado-virt

3.11.1. Currently only demonstration of the Avocado flexibility

3.11.2. using git version 1) git clone $avocado 2) git clone $avocado-virt 3) git clone $avocado-virt-tests cd avocado/avocado ln -s ../../avocado-virt/avocado/virt virt cd - cd avocado/avocado/plugins ln -s ../../avocado-virt/avocado/plugins/virt.py virt.py ln -s ../../avocado-virt/avocado/plugins/virt_bootstrap.py cd -

3.11.3. using installed version python setup.py install yum install avocado-virt

3.11.4. ./scripts/avocado virt-bootstrap Downloads JeOS Check permissions

3.11.5. ./scripts/avocado run ../avocado-virt-tests/qemu/boot.py

3.11.6. ./scripts/avocado run ../avocado-virt-tests/qemu/migration/migration.py

3.11.7. ./scripts/avocado run ../avocado-virt-tests/qemu/usb_boot.py

3.11.8. Qemu templates ./scripts/avocado run DEVCONF/avocado-virt/boot_lspci.py --show-job-log ./scripts/avocado run DEVCONF/avocado-virt/boot_lspci.py --show-job-log --qemu-template DEVCONF/avocado-virt/basic.tpl

4. Links:

4.1. Avocado homepage: http://avocado-framework.github.io/

4.2. Data for this presentation: https://www.dropbox.com/sh/1ucmenbf76vnmn5/AAB4pn157uOOn916JDg2yYzKa?dl=0

4.3. Recording: https://www.youtube.com/watch?feature=player_detailpage&v=rj3XMTS-fwE