A code-searching tool similar to ack, but faster.

Overview

The Silver Searcher

A code searching tool similar to ack, with a focus on speed.

Build Status

Floobits Status

#ag on Freenode

Do you know C? Want to improve ag? I invite you to pair with me.

What's so great about Ag?

  • It is an order of magnitude faster than ack.
  • It ignores file patterns from your .gitignore and .hgignore.
  • If there are files in your source repo you don't want to search, just add their patterns to a .ignore file. (*cough* *.min.js *cough*)
  • The command name is 33% shorter than ack, and all keys are on the home row!

Ag is quite stable now. Most changes are new features, minor bug fixes, or performance improvements. It's much faster than Ack in my benchmarks:

ack test_blah ~/code/  104.66s user 4.82s system 99% cpu 1:50.03 total

ag test_blah ~/code/  4.67s user 4.58s system 286% cpu 3.227 total

Ack and Ag found the same results, but Ag was 34x faster (3.2 seconds vs 110 seconds). My ~/code directory is about 8GB. Thanks to git/hg/ignore, Ag only searched 700MB of that.

There are also graphs of performance across releases.

How is it so fast?

  • Ag uses Pthreads to take advantage of multiple CPU cores and search files in parallel.
  • Files are mmap()ed instead of read into a buffer.
  • Literal string searching uses Boyer-Moore strstr.
  • Regex searching uses PCRE's JIT compiler (if Ag is built with PCRE >=8.21).
  • Ag calls pcre_study() before executing the same regex on every file.
  • Instead of calling fnmatch() on every pattern in your ignore files, non-regex patterns are loaded into arrays and binary searched.

I've written several blog posts showing how I've improved performance. These include how I added pthreads, wrote my own scandir(), benchmarked every revision to find performance regressions, and profiled with gprof and Valgrind.

Installing

macOS

brew install the_silver_searcher

or

port install the_silver_searcher

Linux

  • Ubuntu >= 13.10 (Saucy) or Debian >= 8 (Jessie)

      apt-get install silversearcher-ag
    
  • Fedora 21 and lower

      yum install the_silver_searcher
    
  • Fedora 22+

      dnf install the_silver_searcher
    
  • RHEL7+

      yum install epel-release.noarch the_silver_searcher
    
  • Gentoo

      emerge -a sys-apps/the_silver_searcher
    
  • Arch

      pacman -S the_silver_searcher
    
  • Slackware

      sbopkg -i the_silver_searcher
    
  • openSUSE

      zypper install the_silver_searcher
    
  • CentOS

      yum install the_silver_searcher
    
  • NixOS/Nix/Nixpkgs

      nix-env -iA silver-searcher
    
  • SUSE Linux Enterprise: Follow these simple instructions.

BSD

  • FreeBSD

      pkg install the_silver_searcher
    
  • OpenBSD/NetBSD

      pkg_add the_silver_searcher
    

Windows

  • Win32/64

    Unofficial daily builds are available.

  • winget

      winget install "The Silver Searcher"
    

    Notes:

    • This installs a release of ag.exe optimized for Windows.
    • winget is intended to become the default package manager client for Windows.
      As of June 2020, it's still in beta, and can be installed using instructions there.
    • The setup script in the Ag's winget package installs ag.exe in the first directory that matches one of these criteria:
      1. Over a previous instance of ag.exe from the same origin found in the PATH
      2. In the directory defined in environment variable bindir_%PROCESSOR_ARCHITECTURE%
      3. In the directory defined in environment variable bindir
      4. In the directory defined in environment variable windir
  • Chocolatey

      choco install ag
    
  • MSYS2

      pacman -S mingw-w64-{i686,x86_64}-ag
    
  • Cygwin

    Run the relevant setup-*.exe, and select "the_silver_searcher" in the "Utils" category.

Building from source

Building master

  1. Install dependencies (Automake, pkg-config, PCRE, LZMA):

    • macOS:

        brew install automake pkg-config pcre xz
      

      or

        port install automake pkgconfig pcre xz
      
    • Ubuntu/Debian:

        apt-get install -y automake pkg-config libpcre3-dev zlib1g-dev liblzma-dev
      
    • Fedora:

        yum -y install pkgconfig automake gcc zlib-devel pcre-devel xz-devel
      
    • CentOS:

        yum -y groupinstall "Development Tools"
        yum -y install pcre-devel xz-devel zlib-devel
      
    • openSUSE:

        zypper source-install --build-deps-only the_silver_searcher
      
    • Windows: It's complicated. See this wiki page.

  2. Run the build script (which just runs aclocal, automake, etc):

     ./build.sh
    

    On Windows (inside an msys/MinGW shell):

     make -f Makefile.w32
    
  3. Make install:

     sudo make install
    

Building a release tarball

GPG-signed releases are available here.

Building release tarballs requires the same dependencies, except for automake and pkg-config. Once you've installed the dependencies, just run:

./configure
make
make install

You may need to use sudo or run as root for the make install.

Editor Integration

Vim

You can use Ag with ack.vim by adding the following line to your .vimrc:

let g:ackprg = 'ag --nogroup --nocolor --column'

or:

let g:ackprg = 'ag --vimgrep'

Which has the same effect but will report every match on the line.

Emacs

You can use ag.el as an Emacs front-end to Ag. See also: helm-ag.

TextMate

TextMate users can use Ag with my fork of the popular AckMate plugin, which lets you use both Ack and Ag for searching. If you already have AckMate you just want to replace Ack with Ag, move or delete "~/Library/Application Support/TextMate/PlugIns/AckMate.tmplugin/Contents/Resources/ackmate_ack" and run ln -s /usr/local/bin/ag "~/Library/Application Support/TextMate/PlugIns/AckMate.tmplugin/Contents/Resources/ackmate_ack"

Other stuff you might like

  • Ack - Better than grep. Without Ack, Ag would not exist.
  • ack.vim
  • Exuberant Ctags - Faster than Ag, but it builds an index beforehand. Good for really big codebases.
  • Git-grep - As fast as Ag but only works on git repos.
  • fzf - A command-line fuzzy finder
  • ripgrep
  • Sack - A utility that wraps Ack and Ag. It removes a lot of repetition from searching and opening matching files.
Issues
  • problems with --ignore

    problems with --ignore

    what's the format of the --ignore pattern? If I do

    ag --ignore '*.min.js' time website/public/vendor/ | less
    

    ag still looks into website/public/vendor/jquery-ui.min.js

    I've tried also '.*.min.js', '.min.js$' etc with no results. I'm on OSX 10.9 with ag compiled through homebrew.

    opened by cvlmtg 30
  • Add support for ~/.agrc default config options

    Add support for ~/.agrc default config options

    I can't live without my default options, so this adds support for ~/.agrc similar to .ackrc

    • Load .agrc configuration file from user's home directory if available.
    • CLI options take precedence over .agrc options
    options 
    opened by schlosna 29
  • Question: Plans for limiting search to file types?

    Question: Plans for limiting search to file types?

    Similar to ack, e.g. limit to Ruby files:

    ag "layout" --rb

    opened by stephenmckinney 23
  • Files with non-english characters are trait as binary files

    Files with non-english characters are trait as binary files

    Files with non-english characters are trait as binary files. Test data set:

    test
    тест
    

    Put this two lines in a file and search for "test".

    opened by vsushkov 19
  • --vimgrep option added, reporting every match on the line

    --vimgrep option added, reporting every match on the line

    Output results like vim's :vimgrep /pattern/g would (it reports every match on the line). Here's a ~/.vimrc configuration example:

    set grepprg=ag\ --vimgrep\ $*
    set grepformat=%f:%l:%c:%m
    

    Then use :grep to grep for something. Then use :copen :cn :cp etc.. to navigate through the matches.

    Ag is a good candidate for vim users as an external grepping tool because of its speed and its --column options to get the cursor on the first matching pattern.

    But what about reporting every match on the line? The --vimgrep option does that. The matches navigation isn't approximate anymore and matches are easy to locate with :cn and :cp.

    Here's an example of the --vimgrep output for a file containing:

    foo bar baz
    bar foo baz foo
    bar baz
    
    % ag --vimgrep foo /tmp/a.txt
    /tmp/a.txt:1:1:foo bar baz
    /tmp/a.txt:2:5:bar foo baz foo
    /tmp/a.txt:2:13:bar foo baz foo
    

    The format is:

    "file":"line number":"column number":"line content"
    

    Grepping tools I know (git grep, ack-grep) don't support this output. I think Ag would be the first, making it more and more efficient.

    Here's an example by image: http://i.imgur.com/dkbCuPr.gif

    Let me know what you think.

    Regards, Yann

    opened by inside 18
  • Print match only

    Print match only

    Is it possible to only print the match?

    Say I have the following file:

    Lorem Ipsum is simply dummy text of the printing and typesetting industry.
    

    If I run ag "L.*m" is there a way to have it only print Lorem and not the entire line?

    opened by zachlatta 16
  • Skip long lines (fixes #189)

    Skip long lines (fixes #189)

    This actually implements the functionality to ignore lines longer than some number of characters (default 2000) and adds an option --long-line-length to change the that number. Tested on OS X Yosemite.

    opened by amfarrell 15
  • grep works but ag doesn't

    grep works but ag doesn't

    This is such a simple issue that I must be doing something wrong. This is on Ubuntu 14.04 64-bit using the version of ag installed by apt-get.

    % grep CRON 2014-07-29.tsv [lots of output] % ag CRON 2014-07-29.tsv [no output]

    The -D options shows: DEBUG: Found user's home dir: /home/jonf DEBUG: Skipping ignore file /home/jonf/.agignore DEBUG: Skipping ignore file DEBUG: Query is CRON DEBUG: PCRE Version: 8.31 2012-07-06 DEBUG: Using 1 workers DEBUG: searching path 2014-07-29.tsv for CRON DEBUG: Skipping ignore file 2014-07-29.tsv/.agignore DEBUG: Skipping ignore file 2014-07-29.tsv/.gitignore DEBUG: Skipping ignore file 2014-07-29.tsv/.git/info/exclude DEBUG: Skipping ignore file 2014-07-29.tsv/.hgignore DEBUG: Skipping svn ignore file 2014-07-29.tsv/.svn/dir-prop-base DEBUG: No match in 2014-07-29.tsv DEBUG: Worker finished.

    opened by nobozo 15
  • --invert-match / -v is broken

    --invert-match / -v is broken

    So I'm guessing not many people use ag with inverted matching. Right now:

    $ echo foo >> sample
    $ echo bar >> sample
    
    $ ag foo sample
    1:foo
    
    $ ag -v foo sample
    1:foo
    
    $ ag -v bar sample
    1:foo
    
    $ ag -v b sample
    1:foo
    2:bar
    
    $ mkdir test
    $ cd test
    $ echo foo >> sample
    
    $ ag -v foo
    ./sample
    
    opened by sjl 15
  • Assumption in ignore.c violated, path_start pointing to garbage memory in some cases

    Assumption in ignore.c violated, path_start pointing to garbage memory in some cases

    This took quite a while to debug, and it only results in incorrect behavior in a tiny number of cases, and it's nondeterministic, depending on the contents of garbage memory, which for some reason, happen to be something which makes the program work correctly. I can only get it to misbehave if I have a gitignore pattern that's very long (even something like "# this line is ignored because it starts with # but it is very long").

    This assumption turns out not to hold in all cases: https://github.com/ggreer/the_silver_searcher/blob/07f83361197593302d7af1e2bb1fa0a6ec8c646e/src/ignore.c#L379

    I'm on OS X Yosemite, running the latest version of ag (I tried a bunch of older versions of ag as well, and the issue goes way back). Here is how to reproduce the bug on my machine (you might need to run it a bunch of times since it's a memory issue, or it may not show up at all on your machine):

    1. mkdir ~/testdir && touch ~/testdir/randomfile
    2. git config --global core.excludesfile ~/.gitignore_global
    3. echo "#########################" > ~/.gitignore_global
    4. ag ~/testdir -l --nocolor --hidden -g . --debug

    Depending on the run, I get output like:

    (looks correct):

    DEBUG: Found user's home dir: /Users/anish
    DEBUG: Loading ignore file /Users/anish/.agignore.
    DEBUG: added ignore pattern .ignoreme/ to root ignores
    DEBUG: Found user's global Git excludesfile: /Users/anish/.gitignore_global
    DEBUG: Loading ignore file /Users/anish/.gitignore_global.
    DEBUG: Query is .
    DEBUG: PCRE Version: 8.37 2015-04-28
    DEBUG: Using 4 workers
    DEBUG: No CPU affinity support.
    DEBUG: No CPU affinity support.
    DEBUG: No CPU affinity support.
    DEBUG: No CPU affinity support.
    DEBUG: searching path /Users/anish/testdir for .
    DEBUG: Skipping ignore file /Users/anish/testdir/.agignore: not readable
    DEBUG: Skipping ignore file /Users/anish/testdir/.gitignore: not readable
    DEBUG: Skipping ignore file /Users/anish/testdir/.git/info/exclude: not readable
    DEBUG: Skipping ignore file /Users/anish/testdir/.hgignore: not readable
    DEBUG: Skipping svn ignore file /Users/anish/testdir/.svn/dir-prop-base
    DEBUG: Worker 0 started
    DEBUG: Worker 1 started
    DEBUG: Worker 2 started
    DEBUG: Worker 3 started
    DEBUG: path_start  filename randomfile
    DEBUG: randomfile not ignored
    DEBUG: match_files: file_search_regex matched for /Users/anish/testdir/randomfile.
    /Users/anish/testdir/randomfile
    DEBUG: Worker 0 finished.
    DEBUG: DEBUG: DEBUG: Worker 1 finished.Worker 2 finished.Worker 3 finished.
    

    or incorrect output like:

    DEBUG: Found user's home dir: /Users/anish
    DEBUG: Loading ignore file /Users/anish/.agignore.
    DEBUG: added ignore pattern .ignoreme/ to root ignores
    DEBUG: Found user's global Git excludesfile: /Users/anish/.gitignore_global
    DEBUG: Loading ignore file /Users/anish/.gitignore_global.
    DEBUG: Query is .
    DEBUG: PCRE Version: 8.37 2015-04-28
    DEBUG: Using 4 workers
    DEBUG: No CPU affinity support.
    DEBUG: Worker 0 started
    DEBUG: No CPU affinity support.
    DEBUG: Worker 1 started
    DEBUG: No CPU affinity support.
    DEBUG: Worker 2 started
    DEBUG: No CPU affinity support.
    DEBUG: searching path /Users/anish/testdir for .
    DEBUG: Worker 3 started
    DEBUG: Skipping ignore file /Users/anish/testdir/.agignore: not readable
    DEBUG: Skipping ignore file /Users/anish/testdir/.gitignore: not readable
    DEBUG: Skipping ignore file /Users/anish/testdir/.git/info/exclude: not readable
    DEBUG: Skipping ignore file /Users/anish/testdir/.hgignore: not readable
    DEBUG: Skipping svn ignore file /Users/anish/testdir/.svn/dir-prop-base
    DEBUG: path_start #
     filename randomfile
    DEBUG: randomfile not ignored
    DEBUG: match_files: file_search_regex matched for /Users/anish/testdir/randomfile.
    /Users/anish/testdir/randomfile
    DEBUG: Worker 0 finished.DEBUG: Worker 1 finished.
    DEBUG:
    DEBUG: Worker 2 finished.Worker 3 finished.
    

    specifically, path_start is pointing to garbage, it just happens to be a "#\n\0" left over from reading the gitignore into memory (notice that there are 25 "#"s; if you increase it to 26, you see two "##"s here, and so on, until you cross an 8 byte boundary or something (on my machine)).

    Looking at the operation of filename_filter, in one instance, base_path is /Users/anish/testdir and path is /Users/anish/testdir (both null terminated). This violates the assumptions made in ignore.c:379: base_path does not always end in /\0. Then base_path = path + 25, and it points past the null character into some garbage.

    This shouldn't be too hard to fix. If someone familiar with the code does it, it'll probably be faster. If nobody gets to it soon, I might take a look.

    opened by anishathalye 15
  • add jenkinsfile

    add jenkinsfile

    adding jenkinsfile to the search by file feature

    opened by avnaf 0
  • Add support for .dart files.

    Add support for .dart files.

    opened by peleteiro 0
  • New branch

    New branch

    Just ignore. Testing...

    opened by djm06294 0
  • Link to documentation in the readme

    Link to documentation in the readme

    :wave: Thanks for all the hard work in this project, I was getting a bit frustrated trying to find any documentation for how ignore works (I want to ignore some directory globs that don't seem to be working from .ignore) and it took a search through the issues to discover the wiki, as almost every Github project has a wiki tab (its the default) with nothing in it.

    To help those people who are a bit scatter brained when it comes to looking for docs and expect there to be a "docs" link somewhere on the webpage or readme, I've added one, suggested improvements obviously welcome :)

    opened by JonRowe 0
  • Convert ANSI srting to utf-8 in arguments.

    Convert ANSI srting to utf-8 in arguments.

    In non-English locales, we often search files with multi-byte strings. But Windows does not convert strings. If you prefer the way to use manifest file, I'll update this PR to switch the way.

        <windowsSettings>
          <activeCodePage xmlns="http://schemas.microsoft.com/SMI/2019/WindowsSettings">UTF-8</activeCodePage>
        </windowsSettings>
    
    opened by mattn 1
  • Added Scoop as an installation method on Windows

    Added Scoop as an installation method on Windows

    I just added Scoop as a possible installation method on the Windows platform.

    opened by ferenczy 0
  • --vimgrep inconsistent. How to return 1 match per line?

    --vimgrep inconsistent. How to return 1 match per line?

    How to make --vimgrep return same results as --novimgrep does, but keep the vim format

    opened by squm 0
  • Search talon files

    Search talon files

    opened by mankoff 0
  • How to sponsor fixing bugs exposed when searching large files?

    How to sponsor fixing bugs exposed when searching large files?

    I really love "ag" and use it every day for work as a developer, but it has limitations when searching big files. I have put together my own fork with many PRs which haven't made it into this repo, and it gets a lot closer. It includes PCRE2 from #1035 but it still has trouble with big files. Any way I can sponsor somebody and how do I go about it?

    Test | Greer | Greer Regex w/PCRE | Gcflymoto | Gcflymoto Regex w/PCRE | Gcflymoto Regex w/PCRE2 ---- | ------ | ---------------------- | --------- | ----------- | ----------- Small File | Pass | Pass | Pass | Pass | Pass Small File.gz | Fail | Fail | Pass | Pass | Pass Small File.bz2 | Fail | Fail | Fail | Fail | Fail Small File.xz | Fail | Fail | Pass | Pass | Pass Big File | Fail | Not Supported | Pass | Not Supported | Fail Big File.gz | Fail | Not Supported | Pass | Not Supported | Fail Big File.bz2 | Fail | Not Supported | Fail | Not Supported | Fail Big File.xz | Fail | Not Supported | Pass | Not Supported | Fail

    • HAVE_FOPENCOKKIE disabled because of https://github.com/ggreer/the_silver_searcher/issues/1349#issuecomment-626418179

    Much thanks to @aswild @napcode @afredd @mwagnell @jtamboli @thebaptiste for providing the PRs!!

    @aswild Add support for libpcre2 (compiles and works) #1035 @aswild fix compile warnings on GCC10 #1398 @aswild search_file: check return code of read for failure #1260 @napcode Fixes overflows for large and compressed files #1221 @afredd Rework is_binary() #204 @mwagnell Segmentation fault when searching an xlsx file with -z option #1349 @jtamboli Include file names & line numbers when searching paths, even if stream is present #1433 @thebaptiste To be able to build with lzma.h missing or zlib.h missing or both mis… #1353

    opened by gcflymoto 3
  • ERR: Unsupported compression type: 3 causing seg fault

    ERR: Unsupported compression type: 3 causing seg fault

    I tried to search the bash manual for the string "-z" which resulted in the following error:

    ERR: Unsupported compression type: 3
    [1]    8200 segmentation fault (core dumped)  ag "-z" bash_manual.txt
    

    I suppose it is to do with -z being an option for ag. It did work with:

    ag -- "-z" bash_manual.txt
    

    This is on Ubuntu 20 using zsh.

    Maybe ag should not treat quoted strings as potential options.

    opened by scowan1995 1
Owner
Geoff Greer
CEO & Co-founder of @Floobits. I like to mess with computers and motorcycles.
Geoff Greer
cloc counts blank lines, comment lines, and physical lines of source code in many programming languages.

cloc Count Lines of Code cloc counts blank lines, comment lines, and physical lines of source code in many programming languages. Latest release: v1.9

null 11.7k Jun 7, 2021
:zap: A new cd command that helps you navigate faster by learning your habits.

z.lua A command line tool which helps you navigate faster by learning your habits ⚡ An alternative to z.sh with windows and posix shells support and v

Linwei 2k Jun 6, 2021
Linux Kernel Manager and Activity Monitor 🐧💻

Linux Kernel Manager and Activity Monitor ?? ?? The kernel is the part of the operating system that facilitates interactions between hardware and soft

Orhun Parmaksız 1.2k Jun 1, 2021
Command-line program to download videos from YouTube.com and other video sites

youtube-dl - download videos from youtube.com or other video platforms INSTALLATION DESCRIPTION OPTIONS CONFIGURATION OUTPUT TEMPLATE FORMAT SELECTION

youtube-dl 95.9k Jun 5, 2021
A fast CSV command line toolkit written in Rust.

xsv is a command line program for indexing, slicing, analyzing, splitting and joining CSV files. Commands should be simple, fast and composable: Simpl

Andrew Gallant 7.2k Jun 5, 2021
Magnificent app which corrects your previous console command.

The Fuck The Fuck is a magnificent app, inspired by a @liamosaur tweet, that corrects errors in previous console commands. Is The Fuck too slow? Try t

Vladimir Iakovlev 62.3k Jun 5, 2021
A new way to see and navigate directory trees : https://dystroy.org/broot

Broot A better way to navigate directories Install Broot Get an overview of a directory, even a big one br -s Notice the unlisted? That's what makes i

Canop 5.4k Jun 6, 2021
:cherry_blossom: A command-line fuzzy finder

fzf is a general-purpose command-line fuzzy finder. It's an interactive Unix filter for command-line that can be used with any list; files, command hi

Junegunn Choi 37.1k Jun 6, 2021
Git repository summary on your terminal

A command-line Git information tool written in Rust Onefetch is a command-line Git information tool written in Rust that displays project information

Ossama Hjaji 3.1k Jun 6, 2021
:duck: DuckDuckGo from the terminal

ddgr ddgr is a cmdline utility to search DuckDuckGo (html version) from the terminal. While googler is extremely popular among cmdline users, in many

Piña Colada 2.1k Jun 5, 2021
A command-line benchmarking tool

hyperfine 中文 A command-line benchmarking tool. Demo: Benchmarking fd and find: Features Statistical analysis across multiple runs. Support for arbitra

David Peter 7.9k Jun 6, 2021
The next gen ls command

LSD (LSDeluxe) Table of Contents Description Screenshot Installation Configuration External Configurations Required Optional F.A.Q. Contributors Credi

Pierre Peltier 5.8k Jun 6, 2021
:mag: Google from the terminal

googler is a power tool to Google (web, news, videos and site search) from the command-line. It shows the title, URL and abstract for each result, whi

Piña Colada 5.2k Jun 6, 2021
Command-line program to download image galleries and collections from several image hosting sites

gallery-dl gallery-dl is a command-line program to download image galleries and collections from several image hosting sites (see Supported Sites). It

Mike Fährmann 3.4k Jun 7, 2021