2014-06-03 Jim Meyering version 2.20 * NEWS: Record release date. 2014-05-30 Jim Meyering grep: fix --max-count=N (-m N) to stop reading after Nth match With --max-count=N (-m N), grep is supposed to stop reading input after it has found the Nth match. However, a recent context- related change made it so grep would always read to end of file. * src/grep.c (prtext): Don't let a negative "out_after" value make "pending" line count negative. * tests/max-count-overread: New test, for this. * tests/Makefile.am (TESTS): Add it. * NEWS (Bug fixes): Mention it. * THANKS: Add names of two recent bug reporters. This bug was introduced by commit v2.18-139-g5122195. Reported by Marc Aldorasi in http://bugs.gnu.org/17640. 2014-05-29 Jim Meyering dfa: fix off-by-one under-allocation from recent change Commit v2.19-10-gc32ff67 mistakenly made this change: -realloc_trans_if_necessary (d, 1); +realloc_trans_if_necessary (d, 0); which led to a heap buffer overflow. * src/dfa.c (dfaexec): Allocate space for one state, as before. 2014-05-28 Norihiro Tanaka dfa: fix bug with regex containing multiple begin/end-line constraints grep -E 'a(b$|c$)' would mistakenly match "aa". * src/dfa.c (dfamust): When resetting 'is' in OR, also reset 'begline' and 'endline' of 'must'. * NEWS (Bug fixes): Mention it. This bug was introduced via commit v2.18-85-g2c94326. Reported by Péter Radics in . 2014-05-26 Norihiro Tanaka dfa: simplify building initial state build_state_zero doesn't need the struct dfa to be initialized, so remove the initialization and simplify. * src/dfa.c (build_state_zero): Remove. (dfaexec): Call realloc_trans_if_necessary and build_state directly. dfa: revert "grep: do not count newline before the start of buffer" This reverts commit 5dc3af2806d21455b818be3f9da26c372e4a7f8d. The previous change renders that commit unnecessary. dfa: do not clear the first state of a transition table If number of DFA states reaches 1024, build_state clears transition tables to save memory. However, the initial state is always used, so clearing it just wastes time. * src/dfa.c (build_state): Do not clear the initial state's transition and failure tables. grep: remove unnecessary argument * src/grep.c (do_execute): Remove argument 'start_ptr'. It's always null. All uses changed. 2014-05-24 Paul Eggert grep: --exclude-dir=FOO/ now ignores the trailing slash Problem reported by Khaled Ziyaeen; see: http://bugs.gnu.org/17481 * NEWS, doc/grep.texi (File and Directory Selection): Document this. * src/grep.c (main): Implement this. * tests/include-exclude: Test this. dist: don't distribute lib/colorize.c 'configure' creates this file, so it shouldn't be distributed; see: http://bugs.gnu.org/17480 * configure.ac (COLORIZE_SOURCE): New macro. Don't use AC_CONFIG_LINKS for lib/colorize.c. * lib/Makefile.am (nodist_libgreputils_a_SOURCES): New macro. (libgreputils_a_SOURCES): Remove colorize.c. (CLEANFILES): Add colorize.c (colorize.c): New rule. 2014-05-23 behoffski maint: uncapitalize first letter of two dfaerror message strings * dfa.c (lex): Make two message strings consistent with all of the others: do not capitalize the first letter of the first word. 2014-05-23 Jim Meyering maint: revert "grep: port mb_next_wc to RHEL 6.5 x86-64" This reverts commit v2.18-148-ga6ae68d. Now that we have gnulib change v0.1-131-g2a045bc, "mbrlen, mbrtowc: fix bug with empty input", this work-around is no longer needed. gnulib: update, for mbrlen/mbrtowc empty input bug fix 2014-05-22 Jim Meyering maint: post-release administrivia * NEWS: Add header line for next release. * .prev-version: Record previous version. * cfg.mk (old_NEWS_hash): Auto-update. version 2.19 * NEWS: Record release date. 2014-05-21 Jim Meyering maint: avoid new false-positive syntax-check failure * cfg.mk (exclude_file_name_regexp--sc_prohibit_doubled_word): Exempt new test file that contains legitimate use of "in in". 2014-05-17 Norihiro Tanaka tests: add test case for newline-count fix * tests/count-newline: New test. * tests/Makefile.am (TESTS): Add it. 2014-05-16 Norihiro Tanaka grep: do not count newline before the start of buffer * src/dfa.c (build_state): When checking whether the previous character was a newline, do not count any newline before the start of the buffer. 2014-05-15 Paul Eggert grep: port mb_next_wc to RHEL 6.5 x86-64 * src/searchutils.c (mb_next_wc): Work around glibc bug 16950; see: https://sourceware.org/bugzilla/show_bug.cgi?id=16950 This bug was masked in the other GNU/Linux tests I made. It was exposed on RHEL 6.5 x86-64, where the compiler (GCC Red Hat 4.4.7-4) happened to use temporaries in a different way. Also see recent changes to the Gnulib documentation in this area: http://lists.gnu.org/archive/html/bug-gnulib/2014-05/msg00013.html tests: port mb-non-UTF8-performance to RHEL 6.5 * tests/mb-non-UTF8-performance (timeout): Use an integer, as 'timeout 1.234' doesn't work in EUC locales. 2014-05-12 Paul Eggert egrep, fgrep: port to Solaris 10 /bin/sh This old shell doesn't grok ${0%/*}; see: http://bugs.gnu.org/17471 * src/Makefile.am (egrep fgrep): Don't assume the shell does substrings. * src/egrep.sh (dir): New var, so that the substring calculation is done only once (which is a small win even with newer shells), and so that the calculation is easier to edit on older shells. 2014-05-10 Jim Meyering maint: NEWS: adjust wording to reflect move * NEWS (Improvements): Correct direction-relative wording, now that the referent is below, not above. maint: NEWS: move "Improvements" to the top * NEWS: Move the small "Improvements" section to precede the longer "Bug fixes" one. gnulib: update submodule to latest, and bootstrap * gnulib: Update submodule. * bootstrap: Update from gnulib. 2014-05-10 Paul Eggert dfa: omit double includes * src/dfa.c: Don't include stddef.h or stdbool.h, as dfa.h includes them already, and it's the same module as we are. Suggested by Aharon Robbins in: http://bugs.gnu.org/17458 dfa: fix bug with \< etc in multibyte locales Problem reported by Stephane Chazelas in: http://bugs.gnu.org/16867 * NEWS: Document the fix. * src/dfa.c (dfaoptimize): Remove any superset if changing from UTF-8 to unibyte, and if the pattern has no backreferences. (dfassbuild): In multibyte locales, treat \< \> \b \B as backreferences in the DFA, since the DFA relies on unibyte tests to check them. (dfacomp): Optimize after building the superset, so that dfassbuild can depend on d->multibyte. A downside is that dfaoptimize must remove supersets that are likely slower than the DFA after optimization, but that's been done in the above-described change. * tests/Makefile.am (XFAIL_TESTS): Remove word-delim-multibyte, since the test works now. tests: add test case for -C 0 change * tests/context-0: New test. * tests/Makefile.am (TESTS): Add it. grep: -A 0, -B 0, -C 0 now output a separator Problem reported by Dan Jacobson in: http://bugs.gnu.org/17380 * NEWS: * doc/grep.texi (Context Line Control): Document this. * src/grep.c (prtext): Output a separator even if context is zero. (main): Default context is now -1, not 0. 2014-05-09 Paul Eggert grep: minor improvements to retry-DFA-superset patch * src/dfasearch.c (EGexecute): Avoid unnecessary test in a context where memrchr cannot return a null pointer. 2014-05-09 Norihiro Tanaka grep: retry DFA superset after matching multiple lines * src/dfasearch.c (EGexecute): Without this patch, the code reverts to KWset when the DFA superset matches multiple lines. However, if the DFA superset matches multiple lines, it most likely also matches a single line, and reverting to KWset means dfafast won't work effectively. Change the code so that it retries the DFA superset immediately after it matches multipline lines. On my platform this improves the performance of "LC_ALL=C grep '\(ab\)cd\1d' k" from 3.48 to 2.14 seconds realtime, where k contains the output of "yes abcdabc | head -50000000". dfa: fix inconsistency in multibyte locales * src/dfa.c (dfaexec): Use the same exit condition in multibyte locales as in unibyte. 2014-05-08 Jim Meyering maint: mark some breakless cases with /* fallthrough */ comment * src/dfa.c (addtok_mb, dfaanalyze): Add comment so that it is clear that the "break" statement is deliberately omitted. 2014-05-08 Paul Eggert dfa: assume C89 for CHAR_BIT * src/dfa.c (CHARBITS): Remove. All uses replaced by CHAR_BIT. (NOTCHAR): Now an enum, since it need not be a macro. dfa: don't assume unsigned int is exactly 32 bits wide Sun C 5.12 (sparc) warns of the potential unportability. * src/dfa.c (charclass_word): New type, for clarity. All relevant uses of 'unsigned' changed. (CHARCLASS_WORD_BITS): Rename from INTBITS. All uses changed. Now an enum, since it needn't be a macro. (CHARCLASS_WORD_MASK): New macro. (CHARCLASS_WORDS): Rename from CHARCLASS_INTS. All uses changed. (setbit, clrbit): Cast 1 to charclass_word, for clarity. (notset, add_utf8_anychar, dfastats): Don't assume unsigned int is exactly 32 bits wide. (dfastate): Don't rely on implementation-defined conversion of greater-than-INT_MAX unsigned to int. Change bit test to resemble tstbit more. maint: fix indenting to pacify 'prohibit_tab_based_indentation' * src/dfa.c: Use spaces and not tabs to indent some lines. grep: simplify and clarify invert-related code * src/grep.c (out_invert, prtext): Use bool for booleans. (prline): Remove unnecessary '!!' on a value that is always 0 or 1. (prtext): Remove last arg NLINESP; use !out_invert instead. All uses changed. Move decls to nearer uses, since we can assume C99 here. Update 'outleft' and 'after_last_match' here; it's simpler. (grepbuf): Compute return value by subtracting new from old 'outleft', rather than by keeping a separate running total. Avoid code duplication by arranging for prtext to be called from one place, not three. 2014-05-08 Norihiro Tanaka grep: improve performance of -v when combined with -L, -l or -q Problem reported by Jörn Hees in: http://bugs.gnu.org/17427 * src/grep.c (grepbuf, grep): When -v is combined with -L, -l, or -q, don't read data unnecessarily after a non-match is found. 2014-05-06 Paul Eggert doc: mention performance changes * NEWS: Discuss recent performance improvements and downgrades. dfa: clarify use of "if" The phrase "Y is true if X" is logically equivalent to "X implies Y", but often "X if and only if Y" was intended. * src/dfa.c, src/dfa.h: Reword to avoid the incorrect use of "if". dfa: minor performance improvement for previous change * src/dfa.c (struct dfa): New member 'fast'. Remove 'has_backref'. All uses changed. 2014-05-06 Norihiro Tanaka dfa: speed up 'dfaisfast' * src/dfa.c (struct dfa): New member 'has_backref'. (addtok_mb): Set it. (dfaisfast): Use it. 2014-05-05 Paul Eggert grep: fix -w match next to a multibyte letter * NEWS: Document this. * src/dfasearch.c, src/kwsearch.c (WCHAR): Remove. (wordchar): New static function. * src/dfasearch.c (EGexecute): * src/kwsearch.c (Fexecute): Use the new functions, so that the code works correctly if a multibyte character adjacent to the match has two or more bytes. * src/search.h, src/searchutils.c (mb_prev_wc, mb_next_wc): New functions. * tests/word-delim-multibyte: Add a test for grep -w (which now passes), and a test for \> (which still fails). The \< test also still fails. grep: improve internal API for multibyte boundary * src/search.h, src/searchutils.c (mb_goback): Rename from is_mb_middle. Omit last arg. Return number of bytes to go back, not just a boolean. All uses changed. * src/dfasearch.c (EGexecute): * src/kwsearch.c (Fexecute): Adjust to API change. * src/kwsearch.c (Fexecute): Eliminate common subexpression. grep: fix encoding-error incompatibilities among regex, DFA, KWset This follows up to http://bugs.gnu.org/17376 and fixes a different set of incompatibilities, namely between the regex matcher and the other matchers, when the pattern contains encoding errors. The GNU regex matcher is not consistent in this area: sometimes an encoding error matches only itself, and sometimes it matches part of a multibyte character. There is no documentation for grep's behavior in this area and users don't seem to care, and it's simpler to defer to the regex matcher for problematic cases like these. * NEWS: Document this. * src/dfa.c (ctok): Remove. All uses removed. (parse_bracket_exp, atom): Use BACKREF if a pattern contains an encoding error, so that the matcher will revert to regex. * src/dfasearch.c, src/grep.c, src/pcresearch.c, src/searchutils.c: Don't include dfa.h, since search.h now does that for us. * src/dfasearch.c (EGexecute): * src/kwsearch.c (Fexecute): In a UTF-8 locale, there's no need to worry about matching part of a multibyte character. * src/grep.c (contains_encoding_error): New static function. (main): Use it, so that grep -F is consistent with plain fgrep when the pattern contains an encoding error. * src/search.h: Include dfa.h, so that kwsearch.c can call using_utf8. * src/searchutils.c (is_mb_middle): Remove UTF-8-specific code. Callers now ensure that we are in a non-UTF-8 locale. The code was clearly wrong, anyway. * tests/fgrep-infloop, tests/invalid-multibyte-infloop: * tests/prefix-of-multibyte: Do not require that grep have a particular behavor for this test. It's OK to match (exit status 0), not match (exit status 1), or report an error (exit status 2), since the pattern contains an encoding error and grep's behavior is not specified for such patterns. Test only that KWset, DFA, and regex agree. * tests/prefix-of-multibyte: Add tests for ABCABC and __..._ABCABC___. 2014-05-04 Paul Eggert dfa: minor simplification * src/dfa.c (parse_bracket_exp): Use enum, not macro, and move var to just the scope it's needed. grep: simplify and fix problems with KWset-DFA agreement patch * src/dfa.c (dfambcache, parse_bracket_exp): Simplify. (mbs_to_wchar, wctok, FETCH_WC, match_anychar, match_mb_charset) (check_matching_with_multibyte_ops, transit_state_consume_1char) (transit_state, dfaexec): Use wint_t, not wchar_t, so that WEOF is treated correctly on platforms where WEOF is not a valid wchar_t value. (ctok, lex): Use int, not unsigned int, for characters, so that EOF is treated more naturally. (parse_bracket_exp): Use NOTCHAR to mark uninitialized char, since FETCH_WC can now set the char to EOF. (lex): Remove unnecessary test for EOF. (parse_bracket_exp, atom): Swap then and else parts, to put the small one first; this is more readable here. * src/searchutils.c (is_mb_middle): Simplify. tests: improve coverage for prefix-of-multibyte * tests/prefix-of-multibyte: Also test the regex version. 2014-05-04 Norihiro Tanaka grep: make KWset and DFA agree about invalid sequences in patterns See: http://bugs.gnu.org/17376 * src/dfa.c (dfambcache): Don't cache invalid sequences, because they can't be represented by wide characters. (dfambcache, mbs_to_wchar): Return WEOF for invalid sequences. (ctok): New global variable. (parse_bracket_exp, atom, match_anychar, match_mb_charset): Don't allow WEOF. (lex): Set 'ctok'. * src/kwsearch.c (Fexecute): * src/searchutils.c (is_mb_middle): Don't check here. * tests/invalid-multibyte-infloop: Adjust to fixed behavior. * tests/prefix-of-multibyte: Add test cases for this bug. 2014-05-03 Jim Meyering maint: make ChangeLog generation more robust * Makefile.am (gen-ChangeLog): Sync changes from GNU coreutils, to ensure exit status is propagated, and to support an optional git-log-fix file. 2014-05-03 Paul Eggert grep: clarify EGexecute slightly * src/dfasearch.c (EGexecute): Change if-then-else to !if-else-then. 2014-05-03 Norihiro Tanaka grep: fix the bug in previous patch. * src/dfasearch.c (EGexecute): Do it. 2014-04-30 Paul Eggert grep: simplify EGexecute further * src/dfa.c, src/dfa.h (dfasuperset): Arg is now const pointer. Now pure. * src/dfasearch.c (EGexecute): Coalesce some duplicate code. Don't worry about memrchr returning NULL when that's impossible. 2014-04-30 Norihiro Tanaka grep: adjust timing back to kwset when dfaisfast is true * src/dfasearch.c (EGexecute): If DFA fails after kwset succeeds, the code doesn't return to kwset until it reaches the end of the buffer or finds a match. Because of this, although some cases speed up, others slow down. Adjust the heuristic for switching to the DFA, so that it is more likely to switch at the right times. 2014-04-30 Norihiro Tanaka grep: simplify superset * src/dfa.h (dfahint): Remove decl. (dfasuperset): New decl. * src/dfa.c (dfahint): Remove. (dfassbuild): Rename from dfasuperset. (dfasuperset): New function. It returns the superset of D. * src/dfasearch.c: Use dfasuperset instead of dfahint, and simplify. dfa: optimize memory allocation * src/dfa.c (epsclosure): get the value of 'visited' from the argument. (dfaanalyze): Define and allocate variable 'visited'. (dfastate): Use not 'insert' but 'merge' to insert positions for state 0 of DFA. 2014-04-29 Norihiro Tanaka kwset: improve performance by inlining tr Without this change, older versions of GCC won't inline 'tr', and this can hurt performance significantly. See: http://bugs.gnu.org/17229#64 * src/kwset.c (tr): Make it inline. 2014-04-27 Jim Meyering gnulib: update to latest * gnulib: This fixes a bug whereby running bootstrap would remove our build-aux/git-log-fix file. 2014-04-27 Paul Eggert kwset: improve performance by inlining more Problem reported by Norihiro Tanaka in . * src/kwset.c (bmexec_trans): Rename from bmexec, and make it inline. (bmexec): New implementation, which calls bmexec_trans. This helps GCC inline more aggressively with the default optimization, and improves performance 25% with the reported benchmark on my host. 2014-04-26 Paul Eggert kwset: speed up by using memchr2 Idea suggested by Eric Blake in: http://bugs.gnu.org/17229#43 * bootstrap.conf (gnulib_modules): Add memchr2. * src/kwset.c: Include stdint.h, for uintptr_t. Include memchr2.h. (struct kwset): New members gc1, gc2, gc1help. (tr): Move earlier, so it can be used earlier. (kwsprep): Initialize struct kwset's new members. (memchr_kwset): Rename from memchr_trans. Combine C and TRANS args into new arg KWSET. All uses changed. Use memchr2 when appropriate. (bmexec): Use new members instead of recomputing their values. Increase advance_heuristic; it's just a guess, but memchr2 probably makes it reasonable to increase it. kwset: improve performance when large Boyer-Moore key doesn't match * src/kwset.c (bmexec): As a heuristic, prefer memchr to seeking by delta1 only when the latter doesn't advance much. dfa: fix index bug in previous patch, and simplify * src/dfa.c, src/dfa.h (dfaisfast): Arg is const pointer. * src/dfa.c (dfaisfast): Simplify, since supersets never contain BACKREF. * src/dfa.h (dfaisfast): Declare to be pure. * src/dfasearch.c (EGexecute): Fix typo that could cause buffer read overrun when !dfafast. Hoist duplicate computation out of an if's then and else parts. 2014-04-26 Norihiro Tanaka grep: speed up for a case to repeat failure in DFA after success in kwset A DFA is typically much faster if it is unibyte and does not set BACKREF. Skip kwset if the DFA is fast. For example: yes abcdabc | head -50000000 >k env LC_ALL=C time -p src/grep -i 'abcd.bd' k This improved real-time from 4.86 to 1.34 s. * src/dfa.c, src/dfa.h (dfaisfast): New function. * src/dfasearch.c (EGexecute): Use it. 2014-04-24 Paul Eggert dfa: fix recently-introduced memory leak Problem reported by Aharon Robbins in: http://bugs.gnu.org/17341 * src/dfa.c (dfasuperset): free after dfafree. misc: fix doc and test bugs re grep -z Problem reported by Stephane Chazelas in: http://bugs.gnu.org/16871 * doc/grep.texi (Usage): Remove incorrect example with -P. * tests/pcre: Improve test so that it actually tests whether \s matches a newline. dfa: minor simplification of dfaexec * src/dfa.c (dfaexec): Streamline updating of returned values. Don't bother to check d->multibyte before updating mbp. Avoid duplicate p > end test. 2014-04-24 Paul Eggert dfa: simplify and be more consistent about MB_CUR_MAX * src/dfa.c (struct dfa): New member 'multibyte', replacing 'mb_cur_max'. All uses changed. Use this new member consistently, instead of sometimes referring to MB_CUR_MAX directly. dfa: fix comment * src/dfa.c (maybe_realloc): Fix comment to match behavior better. 2014-04-24 Norihiro Tanaka grep: skip checking of multibyte character boundary, reaching at eolbyte * src/dfa.c (dfaexec): Skip checking of multibyte character boundary, reaching at eolbyte. 2014-04-24 Paul Eggert dfa: fix incorrect comment that led to heap overrun * dfa.c (maybe_realloc): Fix comment to match behavior. dfa: minor tuneup of dfamust memory savings patch * src/dfa.c (allocmust): Use xmalloc, not xzalloc. Initialize the must completely, so that the caller need not invoke resetmust. All callers changed. (dfamust): Omit asserts that aren't needed on typical machines where dereferencing NULL dumps core. Don't leak memory if the pattern contains a NUL byte. 2014-04-24 Norihiro Tanaka grep: avoid wasting memory for large patterns in dfamust * src/dfa.c (struct must): New member 'prev'. It points to the previous must. (allocmust): New function. (freemust): New function. (dfamust): Use it. 2014-04-24 Jim Meyering grep: fix new heap write buffer overrun * src/dfa.c (parse_bracket_exp): Fix off-by-one allocation error. Exposed by running the tests with an ASAN-enabled binary (i.e., created using gcc's -fsanitize=address option). Introduced by commit v2.18-70-gd3d9612, "dfa: simplify range char allocation". 2014-04-24 Paul Eggert build: suppress unsafe-loop-optimizations warnings I ran into one of these while trying out GCC 4.9.0's new -fsanitize=undefined option. The warning told me that GCC didn't do an unsafe optimization, but in 'grep' this is not typically a symptom of a programming error. * configure.ac (WERROR_CFLAGS): Suppress -Wunsafe-loop-optimizations. 2014-04-23 Paul Eggert dfa: fix memory leak reintroduced by previous patch Reported by Norihiro Tanaka in . * src/dfa.c (dfaexec): Allocate mb_match_lens and mb_follows only if not already allocated. (free_mbdata): Null out mb_match_lens to mark it as being freed. 2014-04-23 Jim Meyering tests: use consistent spelling for locale name, en_US.UTF-8 * tests/pcre-infloop: Spell locale name, en_US.UTF-8, consistently, converting this one use from "en_US.utf8", which would provoke a test failure on OS/X. 2014-04-23 Paul Eggert dfa: omit static variables that limited dfaexec to one struct dfa Problem reported by Aharon Robbins in: http://bugs.gnu.org/17328 * src/dfa.c (struct dfa): New member mbs. mb_follows is now a position_set, not a pointer to one; this simplifies memory allocation. All uses changed. (mbs_to_wchar): Put DFA arg at the end, in place of the mbstate_t *arg, since the DFA now contains an mbstate_t. All uses changed. (mbs): Remove static variable. (dfaexec): Remove static bool that attempted to optimize memory allocation, as this wasn't correct for Gawk. Perhaps we can think of a better way to optimize memory. 2014-04-22 Paul Eggert kwset: simplify and speed up Boyer-Moore unibyte -i in some cases This improves the performance of, for example, yes jjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjj | head -10000000 | grep -i jk in a unibyte locale. * src/kwset.c (memchr_trans): New function. (bmexec): Use it. Simplify the code and remove some of the confusing gotos and breaks and labels. Do not treat glibc memchr as a special case; if non-glibc memchr is slow, that is lower priority and I suppose we can try to work around the problem in gnulib. 2014-04-22 Norihiro Tanaka grep: speed-up by using memchr() in Boyer-Moore searching memchr() of glibc is faster than seeking by delta1 on some platforms. When there is no chance to match for a while, use it on them. * src/kwset.c (bmexec): Use memchr() in Boyer-Moore searching. 2014-04-22 Paul Eggert kwset: simplify Boyer-Moore with unibyte -i This change doesn't significantly affect performance on my platform, and should make the code easier to maintain. * src/kwset.c (BM_DELTA2_SEARCH, LAST_SHIFT, TRANS): Remove these macros, in favor of ... (tr, bm_delta2_search): New functions. All uses changed. The latter function is inline because this improves code size and runtime CPU slightly on x86-64 with gcc -O2 (GCC 4.9.0). (bmexec): Prefer tr when that's simpler. 2014-04-22 Norihiro Tanaka grep: may also use Boyer-Moore algorithm for case-insensitive matching * src/kwset.c (BM_DELTA2_SEARCH, LAST_SHIFT, TRANS): New macro. (bmexec): Use character translation table. (kwsexec): Call bmexec for case-insensitive matching. (kwsprep): Change the `if' condition. 2014-04-21 Paul Eggert grep: -P now rejects invalid input sequences in UTF-8 locales See and . * NEWS: Document this. * src/pcresearch.c (Pexecute): Do not use PCRE_NO_UTF8_CHECK, as this leads to undefined behavior when the input is not UTF-8. * tests/pcre-infloop, tests/pcre-invalid-utf8-input: Exit status is now 2, not 1, when grep -P is given invalid UTF-8 data in a UTF-8 locale. dfa: minor improvements to previous patch * src/dfa.c (dfamust): Use &=, not if-then. * src/dfa.h (struct dfamust): * src/dfasearch.c (begline, hwsmusts): Use bool for boolean. * src/dfasearch.c (kwsmusts): * src/kwsearch.c (Fcompile): Prefer decls after statements. * src/dfasearch.c (kwsmusts): Avoid conditional branch. * src/kwsearch.c (Fcompile): Unify the two calls to kwsincr. 2014-04-21 Norihiro Tanaka grep: speed-up for exact matching with begline and endline constraints. dfamust turns on the flag when a state exactly matches the proposed one. However, when the state has begline and/or endline constraints, turns off it. This patch enables to match a state exactly, even if the state has begline and/or endline constraints. If a exact string has one of their constrations, the string adding eolbyte to a head and/or foot is pushed to kwsincr(). In addition, if it has begline constration, start searching from just before the position of the text. * src/dfa.c (variable must): New members `begline' and `endline'. (dfamust): Consideration of begline and endline constrations. * src/dfa.h (struct dfamust): New members `begline' and `endline'. * src/dfasearch.c (kwsmusts): If a exact string has begline constration, start searching from just before the position of the text. (EGexecute): Same as above. * src/kwsearch.c (Fexecute): Same as above. 2014-04-20 Paul Eggert dfa: fix bug that caused NUL to be mishandled in patterns This bug was introduced in the early-2012 patches that fixed some context-handling bugs. Bisecting found commit d8951d3f4e1bbd564809aa8e713d8333bda2f802 (2012-02-05 18:00:43 +0100), but it apears the underlying problem was introduced in commit 8b47c4cf6556933f59226c234b0fe984f6c77dc7 (2012-01-03 11:22:09 +0100). * NEWS: Mention bug fix. * src/dfa.c (char_context): Consider NUL to be a newline only if -z. * tests/Makefile.am (TESTS): Add null-byte. * tests/null-byte: New file. 2014-04-19 Jim Meyering build: reenable some compiler warning options 2014-04-18 Paul Eggert dfa: fix pointer type conversion bug The code converted between size_t * and ptrdiff_t *, which wasn't diagnosed by modern x86-64 GCC but isn't portable. Problem reported by Norihiro Tanaka in . * configure.ac (WERROR_CFLAGS): Don't add -Wno-pointer-sign. We want GCC to diagnose pointer signedness problems, as they violate the C standard and other compilers no doubt complain too. * src/dfa.c (struct dfa): Change type of salloc to size_t. (realloc_trans_if_necessary): Convert signed value to size_t before passing its address to x2nrealloc. Changing the type of tralloc to size_t might have led to problems elsewhere. 2014-04-18 Jim Meyering maint: Revert "dfa: avoid new NULL dereference" This reverts commit 5190041fe515743ef4545abf287d243bc025c701. It was only a bug if one neglected to update to the latest gnulib. With the newer xn2realloc, there is no problem. dfa: avoid new NULL dereference * src/dfa.c (dfa_charclass_index): Restore a "+ 1" mistakenly omitted during recent improvements. Introduced in v2.18-66-g6a60fd5. 2014-04-17 Paul Eggert dfa: minor cleanup * src/dfa.c (MAX): Remove; no longer used. 2014-04-17 Norihiro Tanaka dfa: speed up by checking multibyte characters on demand If dfaexec() runs in non-UTF8 locales, length and wide character representation are checked for all characters of a line in a input string. However, if matched early in the line, results for remaining characters are wasted. This patch checks multibyte characters on demand. It should work faster for early matches, and reduces memory requirements. * src/dfa.c (struct dfa): Remove members mblen_buf, nmblen_buf, inputwcs, ninputwcs. All uses removed. (buf_begin, buf_end, prepare_wc_buf): Remove. All uses removed. (SKIP_REMAINS_MB_IF_INITIAL_STATE): Remove. This is now expanded when used. (match_anychar, match_mb_charset, check_matching_with_multibyte_ops): New arg wc, mbclen. Remove arg idx. All uses changed. (transit_state_consume_1char): New arg wc. All uses changed. (transit_state): New arg 'end'. All uses changed. 2014-04-17 Paul Eggert dfa: trans reallocation microoptimization * src/dfa.c (realloc_trans_if_necessary): Help the compiler avoid unnecessary reloads. dfa: simplify dfmust initialization * src/dfa.c (dfamust): Don't initialize musts twice. Use zcalloc, not xmalloc followed by zeroing. Make result a const pointer. dfa: simplify freelist * src/dfa.c (freelist): Don't null out array while freeing its pointers; the caller can do that if needed. (resetmust): Null out zeroth entry of array. dfa: avoid duplicate strlen when allocating memory * src/dfa.c (dfamust): Use xstrdup, not strlen (twice) + xmemdup. dfa: simplify memory allocation * src/dfa.c (icatalloc, freelist, enlist, comsubs, addlists, inboth) (dfamust): Don't worry about null arguments or results, as memory allocators no longer can return null pointers. (dfamust): Invoke malloc just once when building a concatenated string. dfa: simplify position set and element count allocation * src/dfa.c (dfaanalyze): Allocation position set info all at one go, and similarly for element count info. dfa: simplify multibyte_prop allocation * src/dfa.c (struct dfa): Simplify by removing nmultibyte_prop; it should always be the same as talloc. All uses changed. dfa: simplify range char allocation * src/dfa.c (struct dfa): Simplify by allocating one array of ranges rather than one for range starts and another for range ends. All uses changed. dfa: simplify transition table allocation * src/dfa.c (struct dfa): Remove member 'realtrans', as it can be computed from 'trans'. All uses changed. (realloc_trans_if_necessary): Move earlier, to avoid a forward decl. Use x2nrealloc to compute new size, rather than doing it by hand, which omits a check for unlikely overflow. (realloc_trans_if_necessary, dfafree): Adjust to the fact that d->trans now might be either NULL, or 1 + the pointer to free. (build_state, build_state_zero): Use realloc_trans_if_necessary instead of duplicating its code. dfa: better size-overflow check * src/dfa.c (dfasuperset): Let xnmalloc do the multiplication, to check for size arithmetic overflow better. dfa: avoid unnecessary work and other initialization * src/dfa.c (dfaanalyze, dfainit): Don't bother allocating when x2nrealloc will do it for us. (dfastate): Allocate grps and labels on the stack, as their size is known at compile time. (build_state): Use xmalloc, not xnmalloc, since the multiplication can be done at compile-time. dfa: clarify memory allocation and port to IRIX This change was prompted by a porting problem: IRIX defines its own MALLOC macro, which clashes with ours. More generally, the MALLOC etc. macros are confusing, as they look like functions but do not have C-function semantics. A functional style makes the code easier to read, and though it lengthens the code a bit here it'll make other simplifications easier. * src/dfa.c (XNMALLOC, XCALLOC, CALLOC, MALLOC, REALLOC): Remove. All uses replaced by xnmalloc etc. (REALLOC_IF_NECESSARY): Remove; all uses replaced by .... (maybe_realloc): New function. (copy, merge): Free and allocate rather than realloc, as we needn't save the contents. 2014-04-14 Jim Meyering tests: detect an infloop-inducing bug in grep -P (pcre-8.35) * tests/pcre-infloop: New test. * tests/Makefile.am (TESTS): Add it. 2014-04-12 Paul Eggert build: update gnulib submodule to latest 2014-04-11 Paul Eggert grep: improvements for the open-CSET patch * src/dfa.c (dfamust): Simplify by removing some duplicate code. Optimize patterns like [aaa] even when not case-folding. Avoid an unnecessary copy of the charclass. 2014-04-11 Norihiro Tanaka grep: open CSET and transform into uppercase when MB_CUR_MAX == 1 In unibyte locales with -i, kwset matching isn't helpful, because dfamust doesn't extract the CSET entries. Fix dmamust so that it does that, and makes it possible to take out a longer fixed string from tokens. * src/dfa.c (dfamust): open CSET and transform into uppercase when MB_CUR_MAX == 1. 2014-04-11 Paul Eggert grep: cleanup for HAS_DOS_FILE_CONTENTS issue While cleaning up the empty-string fix, I noticed that one part of the code worried about CRLF in pattern files whereas another part did not. Fix this by using the same approach in both places, and make the CRLF code more modular in the process. * src/dosbuf.c (dos_binary, dos_unix_byte_offsets): New functions. (undossify_input, dossified_pos): Do nothing if ! O_BINARY. * src/grep.c: Always include dosbuf.c so that the code is checked statically even on non-DOS hosts. (dos_binary, dos_unix_byte_offsets): New decls. (undossify_input): Declare unconditionally. * src/grep.c (fillbuf, print_line_head, main): * src/kwsearch.c (Fcompile): Simplify by not worrying about HAVE_DOS_FILE_CONTENTS. * src/grep.c (main): fopen with "rt" if O_TEXT; this is simpler than worrying about HAVE_DOS_FILE_CONTENTS elsewhere. * src/system.h (HAVE_DOS_FILE_CONTENTS): Remove. grep: cleanup for empty-string fix * NEWS: Document it. * src/dfasearch.c (GEAcompile): * src/kwsearch.c (Fcompile): Use C99-style decls to simplify. Avoid duplicate code. * tests/empty-line: Add some more tests like this. 2014-04-11 Norihiro Tanaka grep: no match for the empty string included in multiple patterns * src/dfasearch.c (EGAcompile): Fix it. * src/kwsearch.c (Fcompile): Fix it. 2014-04-08 Paul Eggert grep: remove bool_bf The extra complexity of this microoptimization wasn't ever much help, and currently it generated bigger code with gcc -O2 (x86-64). * src/dfa.c (bool_bf): Remove. All uses replaced by plain 'bool', without a bitfield. 2014-04-08 Jim Meyering maint: avoid sc_po_check syntax-check failure (kwset.c) * po/POTFILES.in: Remove kwset.c from this list, since it no longer contains a translatable diagnostic. 2014-04-08 Paul Eggert grep: port better to hosts with nonstandard nl_langinfo On some hosts, nl_langinfo returns strings other than "UTF-8" when UTF-8 is used, and (worse) return "UTF-8" even if the encoding is single-byte. Work around these problems by trying a sample character instead. * src/dfa.c, src/pcresearch.c, src/searchutils.c: Don't include . * src/dfa.c (using_utf8): Test for UTF-8 by trying a character rather than by invoking nl_langinfo (CODESET); this is more portable in practice, and removes a dependency on HAVE_LANGINFO_CODESET. * src/pcresearch.c: Include dfa.h, for using_utf8. (Pcompile): Use using_utf8 rather than nl_langinfo. 2014-04-07 Paul Eggert grep: prefer bool in DFA internals * src/dfa.c (bool_bf): New type. (dfa_state): Use it, as this seems to generate slightly better code with GCC. (struct mb_char_classes, struct dfa, equal, case_fold, dfasyntax) (laststart, parse_bracket_exp, lex, dfaparse, dfaanalyze, dfastate) (match_mb_charset, dfamust): Use bool for boolean. (using_utf8) [!HAVE_LANGINFO_CODESET]: Tune. (dfaanalyze): Prefer & to && and | to || on booleans; it's simpler here. (dfastate): Simplify charclass nonzero testing. Redo has_mbcset test so that the compiler's more likely to optimize it. 2014-04-07 Norihiro Tanaka grep: prefer regex to DFA for ANYCHAR in multibyte locales * src/dfa.c (dfa_state): New member has_mbcset. Rename backref to has_backref, and make it of type bool too. All uses changed. (state_index, dfastate): Initialize new member. (dfaexec): Prefer regex to DFA for ANYCHAR in multibyte locales. 2014-04-07 Paul Eggert grep: remove trival_case_ignore This optimization is no longer needed, given the other optimizations recently installed. Derived from a patch by Norihiro Tanaka; see . * bootstrap.conf (gnulib_modules): Remove assert-h. * src/dfa.c (CASE_FOLDED_BUFSIZE): Move here from dfa.h. Remove now-unnecessary static assert. (case_folded_counterparts): Now static. * src/dfa.h (CASE_FOLDED_BUFSIZE, case_folded_counterparts): Remove decls; no longer public. * src/dfasearch.c (kwsmusts): Use kwset even if fill MB_CUR_MAX > 1 and case-insensitive. * src/grep.c (MBRTOWC, WCRTOMB): Remove. (fgrep_to_grep_pattern): Use mbrtowc, not MBRTOWC. (trivial_case_ignore): Remove; this optimization is no longer needed. All uses removed. grep: simplify memory allocation in kwset * src/kwset.c: Include kwset.h first, to check its prereqs. Include xalloc.h, for xmalloc. (kwsalloc): Use xmalloc, not malloc, so that the caller need not worry about memory allocation failure. (kwsalloc, kwsincr, kwsprep): Do not worry about obstack_alloc returning NULL, as that's not possible. (kwsalloc, kwsincr, kwsprep, bmexec, cwexec, kwsexec, kwsfree): Omit unnecessary conversion between struct kwset * and kwset_t. (kwsincr, kwsprep): Return void since memory-allocation failure is not possible now. All uses changed. * src/kwset.h: Include , for size_t, so that this include file doesn't require other files to be included first. grep: minor cleanups for Galil speedups * src/kwset.c: Update citations. Include stdbool.h. (kwsincr, kwsprep): Clarify by using C99 decls after statements. (kwsprep): Clarify by using MIN. Avoid a couple of buffer copies when !TRANS. (bmexec): Use bool for boolean. Prefer "continue;" to ";". 2014-04-07 Norihiro Tanaka grep: use the Galil rule for Boyer-Moore algorithm in KWSet The Boyer-Moore algorithm is O(m*n), which means it may be much slower than the DFA. Its Galil rule variant is O(n) and increases efficiency in the typical case; it skips sections that are known to match and does not compare more than once for a position in the text. To use the Galil rule, look for the delta2 shift at each position from the trie instead of the 'mind2' value. * src/kwset.c (struct kwset): Replace member 'mind2' with 'shift'. (kwsprep): Look for the delta2 shift. (bmexec): Use it. 2014-04-06 Paul Eggert grep: cleanup DFA superset optimization * src/dfa.c (dfa_charclass_index): New function, with body of old dfa_charclass but with an extra parameter D. (charclass_index): Reimplement in terms of dfa_charclass_index. (dfahint): Clarify. (dfasuperset): Do not assign to 'dfa' static variable. Instead, use a local, and use the new dfa_charclass_index function. This doesn't fix any bugs, but it's clearer. Initialize a few more members, to simplify dfafree. Copy the charclasses with just one memcpy call. Don't assign nonnull to D->superset until it's known to be valid; that's simpler. (dfafree, dfaalloc): Simplify based on dfasuperset initializations. * src/dfa.h (dfahint): Add comment. * src/dfasearch.c (EGexecute): Simplify use of memchr. Simplify by using memrchr. Fix typo that could cause a buffer read overrun. 2014-04-06 Norihiro Tanaka grep: optimization with the superset of DFA The superset of a DFA is like the DFA, except that for speed ANYCHAR, MBCSET and BACKREF are replaced by (CSET full bits) STAR, and mb_cur_max is 1. For example, for 'a\(b\)c\1': original: a b CAT c CAT BACKREF CAT superset: a b CAT c CAT CSET STAR CAT (The CSET has all bits set.) If a string matches a DFA, it matches the DFA's superset. Using the superset to filter can dramatically improve performance, over 200x in some cases. See . * src/dfa.c (struct dfa): New member 'superset'. (dfahint, dfasuperset): New functions. (dfacomp): Create and analyze the superset. (dfafree): Free only non-NULL items. (dfaalloc): Initialize superset member. (dfaoptimize): If succeed in optimization for UTF-8 locale, don't use the superset. * src/dfa.h (dfahint): New decl. * src/dfasearch.c (EGexecute): Use dfahint. 2014-04-06 Jim Meyering build: avoid OS X 10.8.5 build failure due to lack of static_assert * bootstrap.conf (gnulib_modules): Add assert-h, to accommodate the new use of static_assert on systems lacking support for that construct. Without this change, compilation of dfa.c failed on OS X 10.8.5 with gcc-4.9.0 20140324. We should be using gnulib's assert-h module, regardless, for its nominal improved portability, since grep includes assert.h and uses assert. 2014-04-05 Norihiro Tanaka grep: fix performance bug with regex in line-by-line mode * src/dfasearch.c (EGexecute): Match line-by-line with regex. 2014-04-05 Paul Eggert grep: minor improvements to previous patch * src/dfa.c (MAX): New macro. (match_anychar, match_mb_charset, transit_state_consume_1char): Use it to simplify assignments. (SKIP_REMAINS_MB_IF_INITIAL_STATE): Prefer != 0 for unsigned. (free_mbdata): Omit an unnecessary 'free'. 2014-04-05 Norihiro Tanaka grep: reuse multibyte DFA buffers in non-UTF8 locales * src/dfa.c (struct dfa): New members 'mblen_buf', 'nmblen_buf', 'inputwcs', 'ninputwcs', 'mb_follows' and 'mb_match_lens'. (mblen_buf, inputwcs): Remove static vars. (SKIP_REMAINS_MB_IF_INITIAL_STATE, match_anychar, match_mb_charset) (transit_state_consume_1char, transit_state, prepare_wc_buf): Use new members instead of global variables. (check_matching_with_multibyte_ops): Use new members instead of new allocation. (dfaexec): Initialize new members. (free_mbdata): Free new members. 2014-04-05 Paul Eggert grep: simplify dfa.c by having it not include mbsupport.h directly * src/mbsupport.h: Remove. * src/Makefile.am (noinst_HEADERS): Remove mbsupport.h. * src/dfa.c, src/grep.c, src/search.h: Don't include mbsupport.h. * src/dfa.c: Include wchar.h and wctype.h unconditionally, as this simplifies the use of dfa.c in grep, and it does no harm in gawk. (setlocale, static_assert): Remove gawk-specific hacks, as gawk now does these itself. (struct dfa, dfambcache, mbs_to_wchar) (is_valid_unibyte_character, setbit_wc, using_utf8, FETCH_WC) (addtok_wc, add_utf8_anychar, atom, state_index, epsclosure) (dfaanalyze, dfastate, prepare_wc_buf, dfaoptimize, dfafree, dfamust): * src/dfasearch.c (EGexecute): * src/grep.c (main): * src/searchutils.c (mbtoupper): Assume MBS_SUPPORT. 2014-04-01 Norihiro Tanaka dfa: avoid re-building a state built previously * src/dfa.c (dfaexec): Avoid to re-build a state built previously. 2014-03-28 Paul Eggert dfa: improve port to freestanding DJGPP Suggested by Aharon Robbins (Bug#17056). * src/dfa.c (setlocale) [!LC_ALL]: Return NULL, not "C", reverting part of a recent change. (using_simple_locale): Return true if setlocale returns null. 2014-03-28 Jim Meyering tests: placate "make syntax-check" re compare arg ordering * tests/euc-mb: Reverse order of arguments to compare. Be consistent in ordering compare arguments: expected followed by actual. 2014-03-28 Paul Eggert dfa: avoid an indirection and port wint_t usage * src/dfa.c (struct dfa): Put mbrtowc_cache directly into struct dfa rather than having a pointer; this saves a malloc and an indirection. All uses changed. (dfambcache): Port to hosts where wint_t * can't be cast to wchar_t *. 2014-03-28 Norihiro Tanaka grep: take mbrtowc_cache into new member of struct dfa When struct dfa more than one are used at the same time, mbrtowc cache may be conflict. So, take mbrtowc_cache into new member of struct dfa, and define each mbrtowc cache for them. * src/dfa.c (struct dfa): New member `mbrtowc_cache'. (dfambcache): Rename from build_mbrtowc_cache. Add dependency on struct dfa. (mbs_to_wchar): Add dependency on struct dfa. (FETCH_WC): Use it. (prepare_wc_buf): Use it. Add dependency on struct dfa. (dfacomp): Call it. (dfafree): Release it. 2014-03-28 Paul Eggert dfa: cache results of mbrtowc for speed Idea suggested by Norihiro Tanaka in Bug#16842. * src/dfa.c (mbrtowc_cache): New static var. (build_mbrtowc_cache, mbs_to_wchar): New functions. (FETCH_WC) [MBS_SUPPORT]: Speed up by using mbs_to_wchar instead of mbrtowc and wctob. (FETCH_WC) [!MBS_SUPPORT]: Rewrite in terms of old FETCH macro. (FETCH): Remove; no longer used. (lex): Simplify by avoiding the need for FETCH. (prepare_wc_buf) [MBS_SUPPORT]: Speed up by using mbs_to_wchar. Simplify the loop. (dfacomp): Initialize the cache. 2014-03-27 Norihiro Tanaka grep: perform the kwset-helping DFA match in narrower range When kwsexec gives us the offset of a potential match, we compute line begin/end and then run the DFA matcher to see if there really is a match on that line. When the beginning of the line, BEG, is not on a multibyte character boundary, advance BEG until it on such a boundary, before running the DFA search. * src/dfasearch.c (EGexecute): As above. Add a comment. * tests/euc-mb: Add a test case that exercises this code. This addresses http://debbugs.gnu.org/17095. 2014-03-26 Jim Meyering maint: fix "make dist" * src/Makefile.am (egrep fgrep): Specify egrep.sh via $(srcdir)/egrep.sh, so non-srcdir builds work once again. 2014-03-26 Paul Eggert dfa: improve port to freestanding DJGPP * src/dfa.c (setlocale) [!LC_ALL]: Return "C", not NULL (Bug#17056). (using_simple_locale): Store setlocale result in a ptr-to-const. egrep, fgrep: improve diagnostics from shell scripts This should fix Bug#17098. * src/Makefile.am (EXTRA_DIST): Add egrep.sh. (egrep fgrep): Depend on egrep.sh and Makefile. Build from new file egrep.sh, as this makes the build process easier to follow. Arrange for $0 to look nicer in subgrep. * src/egrep.sh: New file. 2014-03-23 Paul Eggert dfa: avoid undefined behavior * src/dfa.c (FETCH_WC, addtok_wc): Don't rely on undefined behavior when converting an out-of-range value to 'int'. (FETCH_WC, prepare_wc_buf): Don't rely on conversion state after mbrtowc returns a special value, as it's undefined for (size_t) -1. (prepare_wc_buf): Simplify test for valid character. grep: fix and simplify grep -iF optimization * src/grep.c (check_any_alphabets): Remove. (fgrep_to_grep_pattern): Fix problems when mbrtowc returns -1 or -2. Simplify a bit. (main): Don't bother optimizing 'grep -iF PAT' when PAT contains no alphabetics; it's so rare it's not worth the complexity. 2014-03-23 Norihiro Tanaka grep: optimization for fgrep with changing the macher to grep macher. fgrep macher is only use kwset engine. However, it's very slow for case-insensitive matching in multibyte locales. And so, if the matcher is fgrep and case-insensitive and keys including any alphabets, change it into grep matcher by escape of keys. OTOH, if keys include no alphabet, turn match_icase flag off. I prepare following string to measure the performance. yes $(printf '%078dm' 0)| head -1000000 | tr 0 a > in A=`printf '\xef\xbc\xa1'` # FULLWIDTH LATIN CAPITAL LETTER A I run three tests with this patch (best-of-5 trials): env LC_ALL=en_US.UTF-8 time -p src/fgrep -i "$A" in real 8.54 user 7.13 sys 1.16 Back out that commit (temporarily), recompile, and rerun the experiment: env LC_ALL=en_US.UTF-8 time -p src/fgrep -i "$A" in real 0.07 user 0.02 sys 0.05 * src/fgrep.c (Gcompile) New function. * src/main.c (check_any_alphabets) New function. (fgrep_to_grep_pattern) New function. (main) Use them. 2014-03-23 Paul Eggert egrep, fgrep: go back to shell scripts Although egrep's and fgrep's switch from shell scripts to executables may have made sense in 2005, it complicated maintenance and recently has caused subtle performance bugs. Go back to the old way of doing things, as it's simpler and more easily separated from the mainstream implementation. This should be good enough nowadays, as POSIX has withdrawn egrep/fgrep and portable applications should be using -E/-F anyway. * po/POTFILES.in: Remove src/egrep.c, src/fgrep.c, src/main.c. * src/Makefile.am (bin_PROGRAMS): Remove egrep, fgrep. (bin_SCRIPTS): New macro. (grep_SOURCES): Move searchutils.c, dfa.c, dfasearch.c, kwset.c, kwsearch.c, pcresearch.c here from libgrep_a_SOURCES. (egrep_SOURCES, fgrep_SOURCES, noinst_LIBRARIES, libgrep_a_SOURCES): Remove. (LDADD): Remove libgrep.a. (egrep, fgrep): New rules. (CLEANFILES): New macro. * src/grep.c: Rename from src/main.c. (usage, setmatcher, main): Simplify, since there's now just one executable. (Gcompile, Ecompile, Acompile, GAcompile, PAcompile, matchers): Move here from the (removed) src/grep.c. (compile_fp_t, execute_fp_t, struct matcher, matchers): Move here from src/grep.h, as they no longer need to be public. (struct matcher.name): Avoid one level of indirection/relocation. (do_execute, main): Fix a performance bug when it was compiled as 'fgrep', due to confusion about which matcher was which. (main): Fix a performance bug with -P, likewise. * src/grep.h (before_options, after_options): Remove. * src/egrep.c, src/fgrep.c, src/grep.c: Remove. dfa: port to freestanding DJGPP (Bug#17056) * src/dfa.c (setlocale) [!LC_ALL]: Define a dummy. 2014-03-16 Jim Meyering tests: avoid false-positive failure on some AMD CPUs * tests/mb-non-UTF8-performance: Avoid false-positive failure when run on certain AMD processors. 2014-03-10 Jim Meyering tests: make a performance-measuring test less system-sensitive Andreas Schwab reported in http://debbugs.gnu.org/16941 that this test would timeout and fail on m68k-suse-linux. Rather than testing absolute duration with a limit tuned to today's hardware, compare performance of grep with LC_ALL=C against that same command using LC_ALL=ja_JP.eucJP. * tests/init.cfg (require_hi_res_time_): New function. * tests/mb-non-UTF8-performance: Rewrite to use it: record absolute duration D of the first (normally much faster) command, and set a timeout of 8*D for the command running in an affected locale. 2014-03-09 Paul Eggert maint: pacify 'make dist' * src/dfa.c (parse_bracket_exp): Reindent with spaces. * src/dfa.h (case_folded_counterparts): Prefix decl with 'extern'. * src/main.c: Don't include assert.h. 2014-03-07 Paul Eggert fgrep: fix case-fold incompatibility with plain 'grep' fgrep converted to lowercase, whereas the regex code converted to uppercase. The resulting behaviors don't agree in offbeat cases like Greek sigmas and Turkish Is. Fix this by changing fgrep to agree with the regex code. * src/kwsearch.c (Fcompile, Fexecute): * src/searchutils.c (kwsinit, mbtoupper): Convert to uppercase, not to lowercase, for compatibility with plain 'grep'. * src/search.h, src/searchutils.c (mbtoupper): Rename from mbtolower, since it now converts to uppercase. All uses changed. * tests/case-fold-titlecase: Add tests for this. grep: fix case-fold mismatches between DFA and regex The DFA code and the regex code didn't use the same semantics for case-folding. The regex code says that the data char d matches the pattern char p if uc (d) == uc (p). POSIX is unclear in this area; the simplest fix for now is to change the DFA code to agree with the regex code. See . * src/dfa.c (static_assert): New macro, if not already defined. (setbit_case_fold_c): Assume MB_CUR_MAX is 1 and that case_fold is nonzero; all callers changed. (setbit_case_fold_c, parse_bracket_exp, lex, atom): Case-fold like the regex code does. (lonesome_lower): New constant. (case_folded_counterparts): New function. (parse_bracket_exp): Prefer plain setbit when case-folding is not needed. * src/dfa.h (CASE_FOLDED_BUFSIZE): New constant. (case_folded_counterparts): New function decl. * src/main.c (trivial_case_ignore): Case-fold like the regex code does. (main): Try to improve comment re trivial_case_ignore. * tests/case-fold-titlecase: Add lots more test cases. 2014-03-06 Paul Eggert build: update gnulib submodule to latest doc: do not overpromise --ignore-case's behavior * NEWS: Omit vague statement about titlecase that could be misinterpreted, and is more trouble than it's worth. * doc/grep.texi: Add @documentencoding. Fix copyright range to use endash not hyphen. (Matching Control): Do not overpromise what --ignore-case will do. Give examples of corner cases where the documentation does not specify behavior. 2014-03-05 Paul Eggert maint: remove differences from gnulib regex code These don't seem to be needed with GCC 4.8.2, and are making maintenance harder. If we need to disable warnings with older compilers, we can add pragmas to the gnulib versions. See . * gl/lib/regcomp.c.diff, gl/lib/regex_internal.c.diff: * gl/lib/regex_internal.h.diff, gl/lib/regexec.c.diff: Remove. * cfg.mk (exclude_file_name_regexp--sc_prohibit_tab_based_indentation): Don't mention gl/* files. 2014-03-03 Paul Eggert grep: fix comment * src/main.c (trivial_case_ignore): Fix comment typo. 2014-03-03 Norihiro Tanaka grep: avoid to add same character to a bracket expression * src/main.c (trivial_ignore_case): Only when uppercase and/or lowercase is different from original character, add it to new pattern. 2014-03-02 Paul Eggert grep: fix some unlikely bugs in trivial_case_ignore * src/main.c (MBRTOWC, WCRTOMB): Reformat as per usual GNU style. (trivial_case_ignore): Don't overrun buffer in the unusual case when a character has both lowercase and uppercase counterparts. Don't rely on undefined behavior when assigning out-of-range value to an 'int'. Simplify by avoiding unnecessary buffer copies. Work even with shift encodings, by using mbsinit to disable the optimization if we are not in the initial state when we replace B by [BCD]. 2014-03-02 Norihiro Tanaka grep: revert removal of trivial_case_ignore Revive trivial_case_ignore function in order to be able to use kwset. * src/main.c (MBRTOWC, WCRTOMB): New macros. (trivial_case_ignore): New function. (main): Use it. 2014-03-02 Norihiro Tanaka grep: optimization of bracket expression for non-UTF8 locales * src/dfa.c (addtok): Replace an MBCSET with a CSET even in non-UTF8 locales, and even when it has individual characters. 2014-03-01 Paul Eggert doc: describe titlecase fix better * NEWS: Document behavior on lowercase text too. Suggested by Eric Blake in . * doc/grep.texi (Matching Control): Specify behavior of -i more precisely. 2014-02-28 Paul Eggert grep: minor tuning for mb_case_map_apply * src/kwsearch.c (mb_case_map_apply): Avoid unnecessary widening of size_t to intmax_t. Avoid unnecessary reinitialization of k. grep: avoid 'inline' when it doesn't matter These days, compilers generally do just fine without advice from users about 'inline', and there's little need for 'static inline', just as there's little need for 'register'. * src/dfa.c (to_uchar): * src/dosbuf.c (guess_type, undossify_input, dossified_pos): * src/main.c (undossify_input): No longer inline. * src/search.h (mb_case_map_apply): Move from here ... * src/kwsearch.c (mb_case_map_apply): ... to here, and make it no longer 'inline'. grep: fix bugs with -i and titlecase * NEWS: Document this. * src/dfa.c (setbit_wc): Simplify. (setbit_c): Remove; no longer used. (setbit_case_fold_c, parse_bracket_exp, atom): Don't mishandle titlecase. For 'atom', this removes the need for the refactoring of Bug#16729. (lex): Use the slower approach only for letters that have a differing case. * tests/case-fold-titlecase: New file. * tests/Makefile.am (TESTS): Add it. grep: remove lint * src/main.c (MBRTOWC, WCRTOMB): Remove no-longer-used macros. 2014-02-28 Norihiro Tanaka grep: remove trivial_case_ignore * src/main.c (trivial_case_ignore): Remove. (main): Remove its use; this optimization is no longer needed. grep: don't match line-by-line for case-insensitive with grep and awk * src/main.c (matcher): Move decl up. (do_execute): With the grep or awk matchers, no need to match line by line. 2014-02-27 Jim Meyering maint: dfa: pass NULL, not 0, as 2nd arg to setlocale * src/dfa.c (using_simple_locale): Use NULL, not 0. 2014-02-27 Paul Eggert * src/dfa.c (prednames): POSIX allows [[:xdigit:]] to match multibyte chars. * src/dfa.c (parse_bracket_exp): Parenthesize. grep: fix multiple bugs with bracket expressions * NEWS: Document this. * src/dfa.c (using_simple_locale): New function. (parse_bracket_exp): Handle bracket expressions like [a-[.z.]] correctly. Don't assume that dfaexec handles expressions like [^a-z] correctly, as they can match multiple characters in some locales. * tests/posix-bracket: New file. * tests/Makefile.am (TESTS): Add it. 2014-02-25 Stephane Chazelas align grep -Pw with grep -w For the -w option, with -P, we used to look for the pattern surrounded by word boundaries. That's different from what grep -w does and what the documentation describes. Now align with grep -w and the documentation by using PCRE look-behind and look-ahead operators to match the pattern if it is not surrounded by word constituents. * src/pcresearch.c (Pcompile): Use (? grep -P: fix it so backreferences now work with -w and -x To implement -w and -x, we bracket the search term with parentheses. However, that set of parentheses had the default semantics of "capturing", i.e., creating a backreferenceable matched quantity. Instead, use (?:...), to create a non-capturing group. * src/pcresearch.c (Pcompile): Use (?:...) rather than (...). * NEWS (Bug fixes): Mention it. * tests/pcre-wx-backref: New file. * tests/Makefile.am (TESTS): Add it. This addresses http://debbugs.gnu.org/16865 2014-02-20 Jim Meyering maint: post-release administrivia * NEWS: Add header line for next release. * .prev-version: Record previous version. * cfg.mk (old_NEWS_hash): Auto-update. version 2.18 * NEWS: Record release date. tests: test for the non-UTF8 multi-byte performance regression Test for the just-fixed performance regression. With a 100-200x differential, it is reasonable to expect that a very slow system will be able to complete the designated task in a few seconds, while with the bug, even a very fast system would exceed the timeout. * tests/mb-non-UTF8-performance: New file. * tests/Makefile.am (TESTS): Add it. * tests/init.cfg (require_JP_EUC_locale_): New function. grep -i: avoid a performance regression in multibyte non-UTF8 locales * src/main.c: Include dfa.h. (trivial_case_ignore): Perform this optimization only for UTF8 locales. This rectifies a 100-200x performance regression in non-UTF8 multi-byte locales like ja_JP.eucJP. The regression was introduced by the 10x UTF8/grep-i speedup, commit v2.16-4-g97318f5. * NEWS (Bug fixes): Mention it. Reported by Norihiro Tanaka in http://debbugs.gnu.org/16232#50 maint: give dfa.c's using_utf8 function external scope * src/dfa.c (using_utf8): Remove "static inline". * src/dfa.h (using_utf8): Declare it. * src/searchutils.c (is_mb_middle): Use using_utf8 rather than rolling our own. 2014-02-20 Paul Eggert tests: test [^^-^] in unibyte locales This is a bug in the current dfa.c, which was reintroduced by the recent reversion from RRI. * tests/unibyte-negated-circumflex: New file. * tests/Makefile.am (TESTS): Add it. * tests/init.cfg (require_unibyte_locale): New function. grep: fix bug with patterns like [^^-~] in unibyte locales * NEWS: Document this. * src/dfa.c (parse_bracket_exp): Escape patterns like [^^-~], or Awk patterns like [\^-\]], so that they are not misinterpreted by the system regex library. Check for system regex failure due to memory exhaustion. 2014-02-17 Jim Meyering maint: post-release administrivia * NEWS: Add header line for next release. * .prev-version: Record previous version. * cfg.mk (old_NEWS_hash): Auto-update. version 2.17 * NEWS: Record release date. 2014-02-17 Paolo Bonzini revert "grep: DFA now uses rational ranges in unibyte locales" The correct course of action for grep is to defer range interpretation to regex, because otherwise you can get mismatches between regexes with backreferences and those without. For example, [A-Z]. will use RRI but ([A-Z])\1 won't, with the confusing result that the first regex won't match a superset of the language described by the second regex. The source of the confusion is that, even though grep's dfa.c was changed to use range checking instead of strcoll, that code is only invoked if dfaexec is called with backref = NULL, and that never happens for grep! In the end, all that's needed for RRI is compiling --with-included-regex, and in that case the patch is almost a no-op. Almost, because there are corner cases that aren't handled correctly (e.g. [a-[.e.]], or regular expressions that include a NUL character), but this can be handled separately. * NEWS: Revert paragraph introduced by commit v2.16-7-g1078b64. * src/dfa.c (parse_bracket_exp): Revert back to regcomp/regexec. 2014-02-16 Mike Frysinger maint: ignore configure.lineno * .gitignore: Add configure.lineno. 2014-02-11 Benno Schulenberg help: remove surplus newline * src/main.c (usage): Remove inconsistent \n introduced by previous patch. 2014-02-10 Benno Schulenberg help: fix a line ending, and use the same word for similar things * src/main.c (usage): Change a stray 'n' to a newline, and use the word "display" for showing version info as for help text. 2014-02-09 Norihiro Tanaka speed up mb-boundary-detection after each preliminary match After each kwsexec or dfaexec match, we must determine whether the tentative match falls in the middle of a multi-byte character. That is what our is_mb_middle function does, but it was expensive, even when most input consisted of single-byte characters. The main cost was for each call to mbrlen. This change constructs and uses a cache of the lengths returned by mbrlen for unibyte values. The largest speed-up (3x to 7x, CPU-dependent) is when most lines contain a match, yet few are printed, e.g., when using grep -v common-pattern ... to filter out all but a few lines. * src/search.h (build_mbclen_cache): Declare it. * src/main.c: Include "search.h". [MBS_SUPPORT] (main): Call build_mbclen_cache in a multibyte locale. * src/searchutils.c [HAVE_LANGINFO_CODESET]: Include . (mbclen_cache): New global. (build_mbclen_cache): New function. (is_mb_middle) [HAVE_LANGINFO_CODESET]: Use it. * NEWS (Improvements): Mention it. 2014-02-01 Jim Meyering maint: use to_uchar function rather than explicit casts * src/system.h (to_uchar): Define function. * src/kwsearch.c (Fexecute): Use to_uchar twice in place of casts. * src/dfasearch.c (EGexecute): Likewise. * src/main.c (prepend_args): Likewise. * src/kwset.c (U): Define in terms of to_uchar. * src/dfa.c (match_mb_charset): Use to_uchar, not an explicit cast. 2014-01-27 Jim Meyering maint: remove vestiges of support for long-disabled --mmap option This option was disabled in March of 2010, and began to elicit a warning in January of 2012. Its time has come. * doc/grep.in.1: Remove mention. * doc/grep.texi: Likewise. * src/main.c (GROUP_SEPARATOR_OPTION, usage, MMAP_OPTION) (long_options, main): Remove all traces. * tests/Makefile.am (check_PROGRAMS): Remove mention of ignore-mmap. * tests/ignore-mmap: Remove file. * NEWS (Maintenance): Mention it. 2014-01-26 Jim Meyering maint: move two local variable declarations * src/dfasearch.c (kwsmusts): Move one declaration down to the point of definition. Move another into the sole scope where it is used. 2014-01-26 Norihiro Tanaka dfasearch: skip kwset optimization when multi-byte+case-insensitive Now that DFA searching works with multi-byte locales, the only remaining reason to case-convert the searched input is the kwset optimization. But multi-byte case-conversion is so expensive that it's not worthwhile even to attempt that optimization. * src/dfasearch.c (kwsmusts): Skip this function in ignore-case mode when the locale is multi-byte. (EGexecute): Now that this code need not handle multi-byte case-ignoring matches, remove the expensive copy/case-conversion code. With no case-converted buffer, there is no longer any need to call mb_case_map_apply, so remove it and associated code. (kwsincr_case): Remove function. Now, every use of this function is equivalent to a use of kwsincr. Replace all uses. * tests/turkish-eyes: Test all of -E, -F and -G. 2014-01-25 Norihiro Tanaka dfa: remove GREP-ifdef'd code in favor of code used by gawk For many years, gawk and grep have used different #ifdef'd bits of code relating to how the DFA matcher matches multibyte characters. Remove the GREP-specific code in favor of the code gawk uses. This permits us to avoid still more cases in which grep must resort to the expensive process of copying/case-converting each input line before matching against a case-converted regexp. * src/dfa.c (parse_bracket_exp, atom): As above. 2014-01-25 Jim Meyering gnulib: update to latest 2014-01-17 Paul Eggert grep: DFA now uses rational ranges in unibyte locales Problem reported by Aharon Robbins in . * NEWS: * doc/grep.texi (Environment Variables) (Character Classes and Bracket Expressions): Document this. * src/dfa.c (parse_bracket_exp): Treat unibyte locales like multibyte. 2014-01-17 Aharon Robbins grep: add undocumented '-X gawk' and '-X posixawk' options See . * src/grep.c (GAcompile, PAcompile): New functions. (const): Use them. 2014-01-10 Pádraig Brady tests: remove superfluous uses of printf * tests/turkish-eyes: Remove unnecessary uses of printf. 2014-01-09 Jim Meyering grep: make --ignore-case (-i) faster (sometimes 10x) in multibyte locales These days, nearly everyone uses a multibyte locale, and grep is often used with the --ignore-case (-i) option, but that option imposes a very high cost in order to handle some unusual cases in just a few multibyte locales. This change gets most of the performance of using LC_ALL=C without eliminating the ability to search for multibyte strings. With the following example, I see an 11x speed-up with a 2.3GHz i7: Generate a 10M-line file, with each line consisting of 40 'j's: yes jjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjj | head -10000000 > k Time searching it for the simple/noexistent string "foobar", first with this patch (best-of-5 trials): LC_ALL=en_US.UTF-8 env time src/grep -i foobar k 1.10 real 1.03 user 0.07 sys Back out that commit (temporarily), recompile, and rerun the experiment: git log -1 -p|patch -R -p1; make LC_ALL=en_US.UTF-8 env time src/grep -i foobar k 12.50 real 12.41 user 0.08 sys The trick is to realize that for some search strings, it is easy to convert to an equivalent one that is handled much more efficiently. E.g., convert this command: grep -i foobar k to this: grep '[fF][oO][oO][bB][aA][rR]' k That allows the matcher to search in buffer mode, rather than having to extract/case-convert/search each line separately. Currently, we perform this conversion only when search strings contain neither '\' nor '['. See the comments for more detail. * src/main.c (trivial_case_ignore): New function. (main): When possible, transform the regexp so we can drop the -i. * tests/turkish-eyes: New file. * tests/Makefile.am (TESTS): Use it. * NEWS (Improvements): Mention it. 2014-01-07 Paul Eggert tests: port Solaris 10 /bin/sh patch back to GNU/Linux Problem reported by Jim Meyering. * tests/bre, tests/ere, tests/spencer1-locale: Prefer re_shell, not re_shell_. * tests/init.sh (re_shell): New var, which is exported instead of re_shell_. Port to Solaris 10 /bin/sh. Problem reported by Dagobert Michelsen in . * tests/bre, tests/ere, tests/spencer1-locale: Prefer re_shell_ to SHELL, if re_shell_ is set. * tests/init.sh (re_shell_): Export if it's used. 2014-01-01 Jim Meyering maint: post-release administrivia * NEWS: Add header line for next release. * .prev-version: Record previous version. * cfg.mk (old_NEWS_hash): Auto-update. version 2.16 * NEWS: Record release date. gnulib: update to latest, for maint.mk fix maint: update copyright dates for 2014 Do that by running "make update-copyright". gnulib: update to latest 2013-12-31 Jim Meyering pcre: use PCRE_NO_UTF8_CHECK properly In order to obtain the behavior we want, i.e., to disable error-on-invalid-UTF-in-input, apply this PCRE option in pcre_exec, not when compiling. * src/pcresearch.c (Pexecute): Use PCRE_NO_UTF8_CHECK here, ... (Pcompile): ...rather than here. * tests/pcre-invalid-utf8-input: Adjust test case to test for this. 2013-12-26 Jim Meyering maint: fix inconsistent spacing in expression * src/main.c (prline): Fix inconsistent spacing in expression: s/ / /. 2013-12-26 behoffski maint: fix a garbled comment * src/dfa.c (XNMALLOC, etc.): Fix garbled comment wording. 2013-12-23 Jim Meyering maint: fix/improve a comment * src/main.c (prline): Replace untrue FIXME comment with one telling how the hard-to-reach code can be exercised. 2013-12-21 Santiago Ruano Rincón pcre: tell grep -P to relax its stance on invalid multibyte chars Do not exit-2 for invalid UTF-8 characters. Just prior to this change, this command would match no lines and fail like this: $ printf 'j\x82\nj\n'|LC_ALL=en_US.UTF-8 grep -P j|cat -A; echo $? grep: invalid UTF-8 byte sequence in input 2 After this change, the same command matches both lines, and succeeds: jM-^B$ j$ 0 * src/pcresearch.c (Pcompile): Use PCRE_NO_UTF8_CHECK, too, and add a comment. * tests/pcre-utf8: Add a test and a comment. This change did not work with Debian unstable pcre-8.31-2 or with some 8.33 and 8.34-based versions, but does work with Fedora 20's 8.33 and with a built-from-latest source library. Based on a patch by Santiago Ruano Rincón. See http://bugs.gnu.org/15758/ 2013-12-21 Jim Meyering tests: avoid FP failure due to exhausted memory * tests/long-line-vs-2GiB-read: Don't declare the test "failed" when running out of memory. In that case, skip it. 2013-12-18 Jim Meyering maint: add comments and split some long lines * src/main.c (do_execute): Add a comment. Split some lines longer than 80 bytes. pcre: avoid a nominal leak * src/pcresearch.c (Pcompile)[HAVE_LIBPCRE && !PCRE_STUDY_JIT_COMPILE]: We would leak "re" if built with HAVE_LIBPCRE but without PCRE_STUDY_JIT_COMPILE. Move the free out one level. maint: indent cpp directives to reflect nesting * src/pcresearch.c: Insert spaces after a few "#", to indent cpp directives to reflect their nesting. grep: handle lines longer than INT_MAX on more systems When trying to exercize some long-line-handling code, I ran these commands: $ dd bs=1 seek=2G of=big < /dev/null; grep -l x big; echo $? grep: big: Invalid argument 2 grep should not have issued that diagnostic, and it should have exited with status 1, not 2. What happened? grep read the 2GiB of NULs, doubled its buffer size, copied the 2GiB into the new 4GiB buffer, and proceeded to call "read" with a byte-count argument of 2^32. On at least Darwin 12.5.0, that makes read fail with EINVAL. The solution is to use gnulib's safe_read wrapper. * src/main.c: Include "safe-read.h" (fillbuf): Use safe_read, rather than bare read. The latter cannot handle a read size of 2^32 on some systems. * bootstrap.conf (gnulib_modules): Add safe-read. * tests/long-line-vs-2GiB-read: New file. * tests/Makefile.am (TESTS): Add it. * NEWS (Bug fixes): Mention it. 2013-11-25 Jim Meyering tests: port to non-GNU sed * tests/multibyte-white-space (utf8_space_characters): The generation of test inputs relied on GNU sed's interpretation of \<, but that is not portable, and caused spurious test failures. Adjust the sed regexp to work on all versions. Reported by Karl Dubost in http://bugs.gnu.org/15953. 2013-11-22 Jim Meyering maint: minor cleanup: xmalloc+strcpy -> xmemdup * src/main.c (main): Replace an xmalloc+strcpy combination with an equivalent use of xmemdup. 2013-11-21 Jim Meyering Paul Eggert dfa: avoid undefined behavior of "1 << 31" * src/dfa.c (charclass): Change type from "int" to "unsigned int". (tstbit): Rather than shifting "1" left to form a mask, shift the LHS bits the right and use "1" as the mask. Also, return bool, rather than "int". (setbit, clrbit, dfastate): Don't shift "1" (aka (int)1) left by 31 bits. Instead, use "1U" as the operand, to avoid undefined behavior. Spotted by gcc's new -fsanitize=undefined. 2013-11-02 Jim Meyering grep: fix regression with -P vs. invalid UTF-8 input * src/pcresearch.c (Pexecute): Don't abort upon unexpected PCRE-specific error code. Explicitly handle PCRE_ERROR_BADUTF8, and change the default to print a diagnostic including the unhandled integer PCRE error code and exit with status 2. * tests/pcre-invalid-utf8-input: New file. * tests/Makefile.am (TESTS): Add it. * NEWS (Bug fixes): Mention it. * THANKS: Update. Reported by Dave Reisner in http://bugs.gnu.org/15758. grep: fix regression involving \s and \S Commit v2.14-40-g01ec90b made \s and \S work with multi-byte characters, but it made it so any use like \s*, \s+, \s?, \s{3} would malfunction in a multi-byte locale. * src/dfa.c (lex): Also reset laststart. * tests/backslash-s-and-repetition-operators: New file. * tests/Makefile.am (TESTS): Add it. * NEWS (Bug fixes): Mention it. * THANKS: Update. Reported by Mirraz Mirraz in http://bugs.gnu.org/15773. 2013-11-01 Jim Meyering maint: NEWS: document a release-related bug fix * NEWS (Bug fixes): Add an entry for a fix pulled from gnulib. 2013-10-26 Jim Meyering build: update gnulib submodule to latest This pulls in a gnulib fix for maint.mk that ensures the procedure described in README-release actually does what we want. Before this change, that procedure resulted in a grep-2.15 tarball that would lead to a grep binary whose --version- reported version number was 2.14.51... rather than the expected 2.15. maint: avoid automake deprecation warning re ACLOCAL_AMFLAGS * Makefile.am (ACLOCAL_AMFLAGS): Don't use this deprecated variable. * configure.ac (AC_CONFIG_MACRO_DIRS): Use this instead. (AUTOMAKE_OPTIONS): Require automake-1.12. maint: post-release administrivia * NEWS: Add header line for next release. * .prev-version: Record previous version. * cfg.mk (old_NEWS_hash): Auto-update. version 2.15 * NEWS: Record release date. 2013-10-25 Paul Eggert build: port to AIX Problem reported by Pavel Kharitonov in . * src/Makefile.am (LDADD): Add $(LIBTHREAD). build: avoid duplicate -funit-at-a-time etc. options * configure.ac (WERROR_CFLAGS): Don't add -fdiagnostics-show-option and -funit-at-a-time, as Gnulib does that for us now, and we're merely piling on duplicats. 2013-10-24 Jim Meyering tests: port more tests to bourne shells with hex-challenged printf * tests/pcre-utf8: Convert the hex \xHH literals for the euro symbol to octal \OOO. * tests/turkish-I: Likewise for "I with dot". * tests/turkish-I-without-dot: Likewise for another Turkish I: U+0131. maint: clean up an ugly 'while' condition * src/main.c (get_nondigit_option): Separate a slightly baroque "while" expression into two separate statements, both inside the loop. 2013-10-23 Jim Meyering tests: port to bourne shells whose printf doesn't grok hex Use octal escapes, not hex, in printf(1) format strings, and in one case, use $AWK's printf so we can continue to use the table of hex values. * tests/char-class-multibyte: Use printf octal escapes, not hex, for portability to shells like dash and Solaris 10's /bin/sh. * tests/backslash-s-vs-invalid-multitype: Likewise. * tests/surrogate-pair: Likewise. * tests/unibyte-bracket-expr: Count in decimal and convert to octal. * tests/multibyte-white-space (hex_printf): New function. Use it in place of printf so we can retain the table of hex digits without hitting the limitation of some bourne shells. Reported by Paul Eggert in http://bugs.gnu.org/15690#11 2013-10-21 Jim Meyering gnulib: update to latest maint: remove now-unused wcscoll module * bootstrap.conf (gnulib_modules): Remove wcscoll; no longer used. 2013-10-20 Paul Eggert build: avoid chatter from Automake 1.14 * configure.ac (AM_INIT_AUTOMAKE): Add subdir-objects. build: port shell pattern to Solaris 10 * configure.ac: Don't use unquoted '^' in a pattern, as this breaks 'configure' on Solaris 10, whose /bin/sh complains about it, which causes 'configure' to exit even before it finds a decent shell. Unix 7th edition shell accepted '^' as an alias for '|'. build: port to platforms that predefine _FORTIFY_SOURCE Problem reported by Brenton Hoff (Bug#15663). * configure.ac (_FORTIFY_SOURCE): Don't define if already defined. This is what Emacs does. 2013-10-20 Jim Meyering build: update gnulib submodule to latest 2013-10-19 Jim Meyering tests: extend the multibyte-white-space test * tests/multibyte-white-space (utf8_space_characters): Add more single-byte whitespace characters. Align RHS hex values and make the sed substitution less rigid, to accommodate. Also, ensure that grep '\S' exits with status 1. maint: update bootstrap to latest from gnulib * bootstrap: Update from gnulib. maint: fix typo in NEWS * NEWS: Fix/improve example commands in most recent entry. The LC_ALL envvar setting goes before grep, not before printf. Don't reference src/ in the second example command, and do specify the locale. 2013-10-09 Jim Meyering tests: add a test for better coverage of some tricky code * tests/spencer1.tests: Add a non-range bracket expression representing the same regexp, to cover the alternate code path, the one that does not require a regcomp/exec call to interpret the regexp. 2013-10-01 Jim Meyering tests: ensure neither \s nor \S matches an invalid multibyte character * tests/backslash-S-vs-invalid-multitype: New file. Prompted by the bug report from Roman at http://savannah.gnu.org/bugs/?40009 * tests/Makefile.am (TESTS): Add it. dfa: fix \s and \S to work for multibyte * src/dfa.c (lex): In multibyte mode, we can't treat \s and \S as we do in single-byte mode. Map them to [[:space:]] and [^[:space:]] respectively, to make the DFA matcher use the regex-matcher for this term. * tests/multibyte-white-space: New file. Test for the bug. * tests/Makefile.am (TESTS): Add it. This bug was introduced with the addition of DFA support for \s and \S in commit v2.5.4-112-gf979ca0. 2013-09-30 Jim Meyering maint: change all references: s/POSIX\.2/POSIX/ There is no longer any point in referring to POSIX.N. POSIX is sufficient. * doc/grep.in.1: As above. * src/main.c (main): Likewise. * tests/file: Likewise. * tests/options: Likewise. * ChangeLog: Likewise. * NEWS: Likewise. * cfg.mk: Update, to match changed NEWS. Inspired by Glenn Golden's suggestion in http://bugs.gnu.org/15486 2013-09-22 Jim Meyering dfa: remove dead disjunct * src/dfa.c (parse_bracket_exp): Remove dead disjunct. At that point, we know MB_CUR_MAX <= 1, so the test, MB_CUR_MAX > 1 && ... is always false. Remove the disjunct. maint: dfa: improve comments and formatting * src/dfa.c (add_utf8_anychar): Correct wording/alignment of a comment. (dfaexec): Add curly braces around multi-line while statement within a "then" block. (ANYCHAR): Clarify comment: "." does not match an invalid UTF8 character. (parse_bracket_exp) Improve comment. 2013-09-08 Jim Meyering dfa: appease a static analyzer, and save 95 stack bytes * src/dfa.c (MAX_BRACKET_STRING_LEN): Rename from BRACKET_BUFFER_SIZE and decrease from 128 to 32. (parse_bracket_exp): Add one byte more than MAX_BRACKET_STRING_LEN to the length of "str" buffer, to avoid appearance that we may store the trailing NUL beyond the end of buffer. A string of length 32 or greater is rejected by earlier processing, so would never reach this code. Addresses http://bugs.gnu.org/15307 2013-09-01 Corinna Vinschen fix Cygwin UTF-16 surrogate-pair handling with -i grep -i would segfault on systems using UTF-16-based wchar_t (Cygwin) when converting an input string containing certain 4-byte UTF-8 sequences to lower case. The conversions to wchar_t and back to a UTF-8 multibyte string did not take surrogate pairs into account. * src/searchutils.c (mbtolower) [__CYGWIN__]: Detect and handle surrogate pairs when converting. * NEWS (Bug fixes): Mention it. * tests/surrogate-pair: New test. * tests/Makefile.am (TESTS): Add it. Reported by: Jim Burwell 2013-08-19 Paul Eggert doc: mention how to use the latest gnulib * README-hacking: Steal some text from coreutils/README-hacking. 2013-08-10 Jim Meyering build: update gnulib-related code * gnulib: Update submodule to latest. * bootstrap: Update from gnulib. * gl/lib/regex_internal.h.diff: Update to reflect gnulib changes. * bootstrap.conf: Partial sync from coreutils. 2013-08-09 Jim Meyering tests: simplify and factor newest test * tests/char-class-multibyte2: Simplify file names. Factor out $e_acute, so that the grep argument representation is ascii (though the value is still UTF8). doc: NEWS: mention the DFA segfault fix * NEWS (Bug fixes): List the DFA segfault fix. 2013-07-05 Paul Eggert Redo comments and white space to better approach GNU style. 2013-07-05 Paolo Bonzini tests: add testcase for previous change * tests/Makefile.am (TESTS): add char-class-multibyte2. * tests/char-class-multibyte2: New file. 2013-07-05 Mike Haertel dfa: fix multibyte character in brackets with repetition Let FOO stand for any multibyte (e.g. CJK character) in the regexp. It turns out the following much simpler regexp: ([^.]*[FOO]){1,2} is sufficient to cause the crash. In the first step of its parsing, DFA transforms regexp from human readable syntax into reverse-polish form. For regexps of the form a{m,n} repeat counts, it simply builds repeated copies of the representation of a, with appropriate inserted CAT and QMARK operators. For the above example with a regexp of the form a{1,2} it would build: QMARK CAT When building repeated copies of RPN representations, additional copies of the RPN representations are made by calling a function copytoks() with arguments consisting of the start position and length of the original copy. The problem is that the current code for copytoks() is simply incorrect. It operates by calling addtok() for each individual token in the source range being copied. But, in the particular case that the token being added is MBCSET, addtok(): (1) incorrectly assumes that the character set being added to be added is the one most (addtok has no argument to indicate which cset is being added, so it just uses the latest one) (2) attempts to do some token sequence expansion into more primitive operators so things like [FOO] are matched efficiently. Both of these assumptions are incorrect in the case that addtok() is being called from copytoks(): (1) is simply not true, and (2) is redundant--the expansion has already been done token sequence being copied, so there is no need to do the expansion again. The correct function to add exactly one token, without further expansion, is addtok_mb(). So here is my proposed fix, which is that copytoks() should never call addtok(), but instead directly call addtok_mb() (which is what addtok() eventually calls). * src/dfa.c (copytoks): Rewrite using addtok_mb directly. 2013-05-28 Jim Meyering maint: align backslashes consistently * tests/Makefile.am: Most backslashes were aligned with TABs, so adjust the few that used spaces to conform. grep -F: avoid an infinite loop with invalid multi-byte search string * src/kwsearch.c (Fexecute): Avoid an infinite loop when processing a fixed (-F) multibyte search string that is an invalid byte sequence in the current locale and that matches the bytes of the input twice on a line. Reported by Daisuke GOTO in http://thread.gmane.org/gmane.comp.gnu.grep.bugs/4773 * tests/invalid-multibyte-infloop: New test. * tests/Makefile.am (TESTS): Add it. * NEWS (Bug fixes): Mention it. 2013-04-18 Paul Eggert * cfg.mk (old_NEWS_hash): Update. doc: document EREs like a{,10} Problem reported by Eric Blake in . * NEWS: Document the bug fix. * doc/grep.in.1: Restore documentation for this feature, but mention that it is a GNU extension. * doc/grep.texi (Fundamental Structure): Mention that this feature is a GNU extension. 2013-04-02 Paul Eggert build: make dfa.c closer to Gawk's * src/dfa.c: Include , not . stddef.h is smaller and is all we need and is portable nowadays. Include and only if MBS_SUPPORT. 2013-01-15 Paul Eggert grep: make dfa.h standalone Problem reported by Aharon Robbins in . * src/dfa.c: Include dfa.h first, so that it's tested standalone. No need to include , since we are in charge of dfa.h and know that it includes . * src/dfa.h: Include and , so that it's standalone. 2013-01-11 Stefano Lattarini build: update gettext version to 0.18.2 * configure.ac (AM_GNU_GETTEXT_VERSION): Update to 0.18.2. This is necessary to have the gettext-provided m4 files to use AC_PROG_MKDIR_P rather than AM_PROG_MKDIR_P. This latter macro, planned to disappear in Automake 1.14, has already been removed in the development version of Automake, so that, without this change, grep fails to bootstrap with bleeding-edge Automake. 2013-01-11 Paul Eggert build: update gnulib submodule to latest 2013-01-11 Stefano Lattarini build: remove redundant use of $(INCLUDES) * lib/Makefile.am (INCLUDES): Remove. Automake automatically adds $(srcdir) and $(top_builddir) to the C preprocessor search path. INCLUDES is deprecated in Automake 1.13 (causing a runtime warning), and will be removed in Automake 1.14. 2013-01-04 Jim Meyering build: update gnulib submodule to latest maint: update all copyright year number ranges Run "make update-copyright". 2012-11-20 Paul Eggert grep: normalize diagnostics * src/pcresearch.c (Pcompile): Use similar format diagnostics as elsewhere, and translate them. 2012-11-19 Paul Eggert grep: diagnose read errors from -f dir, porting to Solaris Problem reported by Dennis Clarke for Solaris 10 in . * src/main.c (main): For -f F, diagnose any read errors encountered when reading F. * tests/Makefile.am (XFAIL_TESTS): Remove grep-dir. * tests/grep-dir: Don't assume that directories cannot be read via fread, as POSIX allows this and it can happen on Solaris. 2012-11-09 Paolo Bonzini pcre: add PCRE-JIT support for grep * NEWS: Document new feature. * src/pcresearch.c [PCRE_STUDY_JIT_COMPILE] (jit_stack): New. [PCRE_STUDY_JIT_COMPILE] (Pcompile): JIT-compile the regular expression and allocate a stack for it. Based on a patch from Zoltan Herczeg. * THANKS: Add Zoltan to the list. 2012-10-24 Paul Eggert build: go back to AC_PROG_CC * configure.ac: Go back to using AC_PROG_CC rather than AC_PROG_CC_STDC, as the latter is obsolescent and the Autoconf bug involving the former has been fixed. 2012-10-24 Jim Meyering build: use AC_PROG_CC_STDC rather than AC_PROG_CC * configure.ac: Use AC_PROG_CC_STDC rather than AC_PROG_CC, to accommodate autoconf-2.69-37+. build: update gnulib submodule to latest 2012-10-23 Eric Blake build: default to --enable-gcc-warnings in a git tree Anyone building from cloned sources can be assumed to have a new enough environment, such that enabling gcc warnings by default will be useful. Tarballs still default to no warnings, and the default can still be overridden with --disable-gcc-warnings. * configure.ac (gl_gcc_warnings): Set default based on environment. 2012-10-03 Jim Meyering maint: factor out STREQ definition * src/main.c (STREQ): Remove definition. * src/pcresearch.c: (STREQ): Likewise. * src/system.h (STREQ): Define it here instead. maint: correct syntax-check failures; adjust NEWS * tests/pcre-utf8: Reverse order of compare arguments. Remove all copyright year numbers except 2012. Use skip_ "diagnostic...", rather than a bare "exit 77". * NEWS: Start with a concise description of the bug. * src/pcresearch.c (STREQ): Define, so that we can... (Pcompile): use STREQ, not strcmp. 2012-10-03 Paolo Bonzini tests: include UTF-8 testcases for grep -P * tests/Makefile.am (TESTS): Add pcre-utf8. * tests/pcre-utf8: New file. 2012-10-03 Petr Pisar pcresearch: set UTF-8 flag correctly for UTF-8 locales Otherwise, Unicode properties (\p{XXX}) do not work with characters outside the 7-bit ASCII character set. * src/pcresearch.c (Pcompile): Look for UTF-8 locales and set PCRE_UTF8 if one is found. 2012-10-03 Jaroslav Škarvada doc: fix a formatting bug in grep.1 template * doc/grep.in.1: Insert .TP before the paragraph describing --dereference-recursive (-R). 2012-10-03 Jim Meyering maint: placate gcc's -Wjump-misses-init warning * src/kwsearch.c (Fexecute): Replace a "goto" and "return" with a simple return statement, eliminating the label, since that was the sole use. * src/dfasearch.c (EGexecute): Likewise. 2012-09-01 Jim Meyering build: update gnulib submodule to latest 2012-09-01 Eric Blake build: work with new glibc when not optimizing Starting with glibc 2.15, the system headers refuse to compile unconditional use of FORTIFY_SOURCE if optimization is disabled but -Werror is in effect. * configure.ac (FORTIFY_SOURCE): Make conditional. 2012-08-19 Jim Meyering maint: post-release administrivia * NEWS: Add header line for next release. * .prev-version: Record previous version. * cfg.mk (old_NEWS_hash): Auto-update. version 2.14 * NEWS: Record release date. 2012-08-07 Jim Meyering build: update gnulib and bootstrap tests: test for bug with -i and ^$ in a multi-byte locale * tests/empty-line-mb: New file. * tests/Makefile.am (TESTS): Add it. grep -i '^$' in a multi-byte locale could report a false match * src/dfasearch.c (EGexecute): Do not match the sentinel "newline" that is appended to each buffer. This bug may sound like a big deal (it certainly surprised me), but realize that only the empty-line-matching regular expression '^$' can trigger it, and then only when you add the unnecessary (and arguably superfluous) -i, *and* run the command in a multi-byte locale. Using a multi-byte locale for such a regular expression is also pointless, and hurts performance. * NEWS (Bug fixes): Mention it. Reported by Alexander Katassonov 2012-08-06 Jim Meyering tests: fix a skip diagnostic that mentioned the wrong locale * tests/init.cfg (require_tr_utf8_locale_): s/en_US/tr_TR/ 2012-08-02 Jim Meyering tests: skip failing test on FS/system that lack SEEK_HOLE support * tests/big-hole: Test for SEEK_HOLE support. If not available, skip this test. Hence, this test is now skipped on linux-3.5.0 with ext4 or tmpfs. The test runs (and passes) with at least btrfs, xfs, or ocfs2. * bootstrap.conf (gnulib_modules): Use the perl module. 2012-07-30 Jim Meyering maint: optimize long-line processing * src/main.c (grep): Use memrchr rather than an open-coded loop, reducing the cost of the replaced code by 50% when processing very long lines. If there were a rawmemrchr function (analogous to glibc's rawmemchr), then the performance improvement would be even greater. 2012-07-27 Paul Eggert maint: remove stat-size * bootstrap.conf (gnulib_modules): Remove stat-size. * src/main.c: Don't include stat-size.h; no longer needed. grep: don't falsely report compressed text files as binary * NEWS: Document this. * src/main.c (file_is_binary): Remove the heuristic based on st_blocks, as it does not work for compressed file systems. On Solaris, it'd be cheap to test whether the file system is known to be uncompressed, which allow the heuristic, but Solaris has SEEK_HOLE so there's little point. grep: don't falsely report tiny text files as binary * NEWS: Document this. * src/main.c (file_is_binary): When we are already at apparent EOF, skip the file-size check, as some servers use zero blocks to store binary files. Reported by Martin Carroll in . 2012-07-26 Paul Eggert doc: document -r/-R in man page * doc/grep.in.1: Document -r vs. -R. 2012-07-21 Jim Meyering tests: avoid false positive upon kernel OOM-kill * tests/big-match (skip_diagnostic): Handle case of 139 (SIGKILL) with no diagnostic. build: update gnulib and bootstrap maint: fix misspellings in old ChangeLog * ChangeLog-2009: Fix typos. 2012-07-19 Paul Eggert grep: fix ptrdiff/size_t clash Reported by Jaroslav Škarvada in . * src/dfasearch.c (EGexecute): Use size_t, not ptrdiff_t, for lengths. Use regoff_t to store re_match's output, and test it before converting it to size_t. 2012-07-06 Jim Meyering maint: correct log typo, to reflect in generated ChangeLog * Makefile.am (gen-ChangeLog): Use --amend, now that we must make our first log correction. * build-aux/git-log-fix: New file. 2012-07-04 Jim Meyering maint: post-release administrivia * NEWS: Add header line for next release. * .prev-version: Record previous version. * cfg.mk (old_NEWS_hash): Auto-update. version 2.13 * NEWS: Record release date. build: update gnulib submodule, bootstrap, init.sh 2012-06-17 Jim Meyering tests: add another turkish-I-related test case * tests/turkish-I-without-dot: Also exercise the case in which the original string and the lower-case buffer have precisely the same length (22 bytes here), yet internal offsets do differ. 2012-06-16 Jim Meyering grep -i: work also when converting to lower-case inflates byte count Commit v2.12-16-g7aa698d addressed the case in which the lower-case representation of an input byte occupies fewer bytes than the original. However, even with commit v2.12-20-g074842d, grep -i would still misbehave when converting a character to lower-case increased its byte count. The map-manipulation code assumed that the case conversion could only shrink the byte count. With the consideration that it may also inflate it, the deltas recorded in the map array must be signed, and we must account for the one-to-two-or-more mapping when the original-to-lower-case conversion causes the byte count to increase. * src/searchutils.c (mbtolower): When a lower-case character occupies more than one byte, set its remaining map slots to zero. Change the type of the map to be signed, and compute the change in character byte count as new_length - old_length. * src/search.h: Include , for decl of intmax_t. (mb_case_map_apply): Adjust for signed increments: each map entry is now signed. (mb_len_map_t): Define type. Thanks to Paul Eggert for noticing in review that using a bare "char" as the base type would be wrong on systems for which it is a signed type (as with gcc's -funsigned-char). * src/kwsearch.c (Fcompile, Fexecute): Likewise. * src/dfasearch.c (kwsincr_case, EGexecute): Likewise. * tests/turkish-I-without-dot: New test. Thanks to Paolo Bonzini for the tip that in the tr_TR.utf8 locale, mapping "I" to lower case increases the character's byte count. * tests/Makefile.am (TESTS): Add it. * tests/init.cfg (require_tr_utf8_locale_): New function. * NEWS (Bug fixes): Expand the existing entry. 2012-06-12 Paul Eggert grep: handle -i when chars differ in length but line does not * src/searchutils.c (mbtolower): Return the map back to the caller if any input character's length differs from the corresponding output character's, not merely if the total string length differs. Problem reported by Johannes Meixner in . 2012-06-07 Jim Meyering tests: extend coverage of dfa.c's match_mb_charset Add a test case to increase test coverage of part of dfa.c (the DFA matcher used by grep and gawk). While thinking about removing the few remaining uses of strncpy in dfa.c, I found that none of the existing tests covered the 40+ lines of code at the end of match_mb_charset, so constructed this test case to demonstrate that it's not dead code * tests/dfa-coverage: New test, for improved coverage. * tests/Makefile.am (TESTS): Add it. 2012-06-05 Jim Meyering build: fix a subtly twisted "make distcheck" failure "make distcheck" would fail when, during a test build, an attempt to overwrite the deliberately-write-protected $(srcdir)/grep.pot file would fail. * bootstrap.conf (bootstrap_epilogue): Don't let the existence of a large sparse file in the build directory induce "make distcheck" failure. The existence of a large sparse test file named 8T-or-so would make po/Makefile.in.in's use of grep (to search for "GNU grep" as an indication that this is a GNU package) exit 2 without generating any output, which made the first xgettext use --package-name=grep, while that same search for "GNU grep" would succeed when run from a pristine from-tarball build, thus making the second xgettext invocation use --package-name='GNU grep'. That mismatch: -"Project-Id-Version: grep 2.12.18-1080\n" +"Project-Id-Version: GNU grep 2.12.18-1080\n" led to the attempt by Makefile.in.in's grep.pot-update rule to overwrite ../../grep.pot in the read-only po/ source directory. 2012-06-03 Jim Meyering build: update gnulib submodule, bootstrap and init.sh cfg.mk: Exempt dfa.c from the new no-strncpy test, for now. 2012-06-02 Jim Meyering grep: fix how -i works with a match containing the Turkish I-with-dot Fix a long-standing problem in the way grep's -i interacts with data whose byte count changes when we convert it to lower case. For example, the UTF-8 Turkish I-with-dot (İ) occupies two bytes, but its lower case analog, i, occupies just one byte. The code converts both search string and the haystack data to lower case, and then searches for the modified string in the modified buffer. The trouble arose when using a lowercase buffer pair to manipulate the original (longer) buffer. The solution is to change mbtolower to return additional information: a malloc'd mapping vector. With that, the caller maps the lowercase- relative to numbers that refer to the original buffer. This mapping is used only when lengths actually differ, so the cost in general should be small. * src/searchutils.c (mbtolower): Add the new map parameter. * src/search.h (mb_case_map_apply): New function. * src/kwsearch.c (Fexecute): Update mbtolower caller, and upon success, apply the new map. * src/dfasearch.c (EGexecute): Likewise. * tests/Makefile.am (XFAIL_TESTS): Remove turkish-I from this list; that test is no longer expected to fail. * NEWS (Bug fixes): Mention it. Reported by Ilya Basin in http://thread.gmane.org/gmane.comp.gnu.grep.bugs/3413 and later by Strahinja Kustudic in http://savannah.gnu.org/bugs/?36567 2012-06-01 Paul Eggert grep: remove unnecessary "what-if-signal?" code * src/main.c (fillbuf): Don't worry about EINTR when closing -- not possible, since we're not catching signals. 2012-05-16 Paul Eggert grep: avoid nominal integer overflow * src/dfa.c (add_utf8_anychar): Avoid signed integer overflow. Although this works on all platforms we know about, strictly speaking the behavior is undefined, and Sun C 5.8 warns about it. 2012-05-15 Jim Meyering maint: avoid nit-picky syntax-check test failure; tweak big-hole test * NEWS: Restore deleted newline in "old" NEWS, to fix a syntax-check test failure. * tests/big-hole: Use awk, rather than a shell loop: saves 3000 lines of verbose shell output in the .log file. 2012-05-15 Paul Eggert grep: sparse files are now considered binary * NEWS: Document this. * doc/grep.texi (File and Directory Selection): Likewise. * bootstrap.conf (gnulib_modules): Add stat-size. * src/main.c: Include stat-size.h. (usable_st_size): New function, mostly stolen from coreutils. (fillbuf): Use it. (file_is_binary): New function, which looks for holes too. (grep): Use it. * tests/Makefile.am (TESTS): Add big-hole. * tests/big-hole: New file. 2012-05-06 Paul Eggert maint: quote 'like this' or "like this", not `like this' See . * ChangeLog-2009, HACKING, NEWS, README-hacking, cfg.mk, configure.ac: * lib/colorize-w32.c, m4/pcre.m4: * src/Makefile.am, src/dfa.c, src/dosbuf.c, src/main.c: * tests/backref, tests/help-version, tests/tests: In commentary, quote 'like this' or "like this" rather than `like this' or ``like this''. * cfg.mk (old_NEWS_hash): Update due to changed old NEWS. * doc/grep.texi (General Output Control): Quote sample text with @samp, not with `...'. * src/main.c (usage): * tests/help-version: Quote 'like this' rather than `like this' in diagnostics. exclude: process exclude and include directives in order Also, change exclude and include directives so that they apply to command-line arguments too. This restores the pre-2.6 behavior, and fixes a bug reported by Quentin Arce in . * NEWS: Document this. * src/main.c (included_patterns): Remove. All uses removed. (skipped_file): New function. (grepdirent): New arg command_line; all callers changed. This is needed because non-command-line files can invoke fts_open, and their directory entries need to be distinguished from top-level directory entries. Move code into the new skipped_file function. (grepdesc): Check whether a command-line argument should be skipped. (main): --include and --exclude options now share excluded_patterns rather than having separate variables included_patterns and excluded_patterns. * tests/include-exclude: Add a test to detect the fixed bug. build: update gnulib submodule to latest 2012-04-30 Jim Meyering cosmetic: binary operator goes *after* the newline, when split * src/dfa.c (match_mb_charset): Join split lines. (parse_bracket_exp): Move "||" from end of first split line to the beginning of the continued line. * src/dosbuf.c (dossified_pos): Likewise, but for "&&". grep: -K is not an option: remove it from list The presence of "K" in the short-option string meant that an erroneous "grep -K ..." would fail with a bare Usage/Try... message, without the usual "invalid option -- 'K'". With this removal, now grep prints the expected invalid option diagnostic. * src/main.c (short_options): Remove "K". Reported by Петр Досычев in http://thread.gmane.org/gmane.comp.gnu.grep.bugs/4488 2012-04-29 Paolo Bonzini dfa: small fixes to single-byte range computation * src/dfa.c (parse_bracket_exp): Do not call regexec with an invalid subject. Move declarations before all statements. 2012-04-27 Paolo Bonzini dfa: do not use hard-locale * bootstrap.conf (gnulib_modules): Remove hard-locale. * src/dfa.c (hard_LC_COLLATE): Remove. (dfaparse): Do not initialize it. (parse_bracket_exp): Always go through system regex matcher to find single byte characters matching a range. drop support for Makefile.boot * Makefile.am: Do not distribute README-boot and Makefile.boot. * NEWS: Mention this change. * README-alpha: Do not mention README-boot and Makefile.boot. * Makefile.boot: Remove. * README-boot: Remove. 2012-04-27 Aharon Robbins dfa: do not use strcoll to match multibyte characters in ranges This does not affect the behavior of grep, which always defers to glibc or gnulib when matching ranges. * src/dfa.c (match_mb_charset): Compare wc directly to the range endpoints. dfa: include stdbool.h explicitly * src/dfa.c: Include stdbool.h explicitly 2012-04-23 Jim Meyering maint: post-release administrivia * NEWS: Add header line for next release. * .prev-version: Record previous version. * cfg.mk (old_NEWS_hash): Auto-update. version 2.12 * NEWS: Record release date. build: update gnulib submodule to latest tests: skip annoyingly long gnulib lock tests * bootstrap.conf (avoided_gnulib_modules): Define. (gnulib_tool_option_extras): Use it. 2012-04-22 Jim Meyering tests: avoid spurious quote-mismatch failure on OS/X * tests/in-eq-out-infloop: Simplify expected error output, eliminating expected quotes altogether, thus avoiding spurious OS/X-specific failure due to mismatch of multi-byte vs. single-byte quotes. 2012-04-17 Jim Meyering build: update gnulib submodule to latest * bootstrap: Also update this file. 2012-04-17 Jim Meyering grep: fix --devices=ACTION (-D) so stdin is once again exempt An oversight in the 2.11 changes made it so "echo x|grep x" would fail for those who set GREP_OPTIONS=--devices=skip. * src/main.c (grepdesc): Ignore skip-related options when reading from standard input. * tests/skip-device: New file. Test for the above. * tests/Makefile.am (TESTS): Add it. * doc/grep.texi (File and Directory Selection): Clarify this point, documenting the stdin exemption. * NEWS (Bug fixes): Mention it, and add a few "[fixed in ...] notes. Reported by Tino Keitel in http://bugs.debian.org/669084, and forwarded to bug-grep by Aníbal Monsalve Salazar. 2012-04-13 Jim Meyering maint: dfa: correct bogus formatting * src/dfa.c (transit_state, dfaexec): s/++ * VAR/++*VAR/ maint: dfa: add/improve comments * src/dfa.c (transit_state_consume_1char): Note always-ignored return value. Fix typos: s/equivalent class/equivalence class/. maint: dfa: avoid unnecessary uses of strcpy/strncpy * src/dfa.c (icatalloc): Use memcpy, not strcpy, given the length. (dfamust): Combine MALLOC+strcpy into cleaner xmemdup. (parse_bracket_exp): Likewise, but replace a use of strncpy. grep: handle symlinked directory loops as usual * src/main.c (grepfile): Treat EMLINK just like ELOOP, for systems like FreeBSD 9.0 on which we would otherwise report "Too many links" rather than ignoring that type of failure. E.g., "mkdir d; cd d; ln -s . a; grep -r ^" would print grep: a: Too many links and would exit with status 2. Now, it prints nothing and exits with status 1, as before. Reported by Nelson H. F. Beebe. tests: avoid spurious failure of the symlink test * tests/symlink: Ignore spurious "Binary file d matches" on systems for which reading from a directory actually succeeds. Reported by Bruno Haible and Nelson Beebe. 2012-04-09 Jim Meyering tests: avoid syntax-check failure: reverse compare arguments * tests/repetition-overflow: Fix reversed compare arguments. build: update gnulib submodule to latest 2012-03-18 Paul Eggert grep: report overflow for ERE a{1000000000} * NEWS: Document this. * src/dfa.c (MIN): New macro. (lex): Lexically analyze the repeat-count operator once, not twice; the double-scan complicated the code and made it harder to understand and fix. Adjust the repeat-count parsing so that it better matches the behavior of the regex code, in three ways: 1. Diagnose too-large repeat counts rather than treating them as literal characters. 2. Use RE_INVALID_INTERVAL_ORD, not RE_NO_BK_BRACES, to decide whether to treat invalid-syntax {...}s as literals. 3. Use the same wording for {...}-related diagnostics that the regex code uses. * tests/bre.tests, tests/ere.tests, tests/repetition-overflow: Adjust to match new behavior, and add a few tests. * cfg.mk (exclude_file_name_regexp--sc_error_message_uppercase): New macro, since the diagnostics start with uppercase letters. 2012-03-14 Paul Eggert grep: -r no longer follows symlinks; use fts Change -r to follow only command-line symlinks, and by default to read only devices named on the command line. This is a simple way to get a more-useful behavior when searching random directories; the idea is to use 'find' if you want something fancy. -R acts as before and gets a new alias --dereference-recursive. The code now uses fts internally, so it is more robust and faster with large hierarchies. * .gitignore: Remove lib/savedir.c, lib/savedir.h. * tests/symlink: New file * Makefile.boot (LIB_OBJS_core): Remove isdir.o, savedir.o. Perhaps other changes are needed too, but I'm not sure what this makefile is for. * NEWS: Document changes. * doc/grep.texi (File and Directory Selection): Likewise. * bootstrap.conf (gnulib_modules): Remove dirent, dirname, isdir, open. Add fstatat, fts, openat-safer. * lib/Makefile.am (libgreputils_a_SOURCES): Remove savedir.c, savedir.h. * lib/savedir.c, lib/savedir.h: Remove. * po/POTFILES.in: Add lib/openat-die.c. * src/main.c: Include fcntl-safer.h, fts_.h. Don't include isdir.h, savedir.h. (struct stats, stats_base): Remove. (long_options, usage, main): Add --dereference-recursive and implement -r vs -R. (filename_prefix_len, fts_options): New static vars. (basic_fts_options, READ_COMMAND_LINE_DEVICES): New constants. (devices): Now defaults to READ_COMMAND_LINE_DEVICES. (reset, grep): Now takes just struct stat rather than file name and struct stats. All callers changed. (fillbuf): Now takes struct stat reather than struct stats. All callers changed. (grep): Don't worry about recursing too deeply; fts and grepdesc handle this now. (is_device_mode, grepdirent, grepdesc, grep_command_line_args): New functions. (grepfile): New args DIRDESC, FOLLOW, COMMAND_LINE. Remove struct stats arg. All callers changed. Use openat_safer rather than open. Use desc == STDIN_FILENO to tell whether we're reading "-". Don't worry about EINTR when closing -- not possible, since we're not catching signals. * tests/Makefile.am (TESTS): Add symlink. * tests/symlink: New file. 2012-03-12 Paul Eggert tests: port big-match to non-GNU dd * tests/big-match: Don't assume GNU dd extension "bs=1M". tests: test for bug with -r --exclude-dir and no file operand * tests/include-exclude: Test for the bug and fix. 2012-03-12 Allan McRae grep: fix segfault with -r --exclude-dir and no file operand * src/main.c (grepdir): Don't invoke excluded_file_name on NULL. * NEWS (Bug fixes): Mention it. 2012-03-09 Jim Meyering tests: exercise two recently-fixed bugs * tests/repetition-overflow: New test for bugs fixed by commit v2.10-82-gcbbc1a4. * tests/Makefile.am (TESTS): Add it. 2012-03-03 Jim Meyering maint: use an optimal-for-grep xz compression setting * cfg.mk (XZ_OPT): Use -6e (determined empirically, see comments). This sacrifices a meager 60 bytes of compressed tarball size for a 55-MiB decrease in the memory required during decompression. I.e., using -9e would shave off only 60 bytes from the tar.xz file, yet would force every decompression process to use 55 MiB more memory. build: update gnulib submodule to latest 2012-03-02 Jim Meyering maint: post-release administrivia * NEWS: Add header line for next release. * .prev-version: Record previous version. * cfg.mk (old_NEWS_hash): Auto-update. version 2.11 * NEWS: Record release date. tests: avoid failure when using Solaris 10's sed * tests/reversed-range-endpoints: Use a simpler sed expression to sanitize actual output, so it also works with Solaris 10's /bin/sed. 2012-03-01 Jim Meyering maint: manually correct formatting in dfa.c's cpp definitions * src/dfa.c: Adjust formatting in cpp definitions. maint: indent dfa.c * src/dfa.c: Filter through indent like this: HOME=. indent -Tsize_t -l79 --leave-preprocessor-space \ --dont-format-comments --no-tabs < dfa.c > k && mv k dfa.c doc: correct grep.1's descriptions of \w and \W (they omitted "_") * doc/grep.in.1: Fix descriptions of \w and \W. They did not mention "_". * doc/grep.texi (The Backslash Character and Special Expressions): [\w, \W]: List the "_" before the char class, not after: [_[:alnum:]], for readability and to be consistent with the man page. 2012-03-01 Paul Eggert maint: spelling fixes grep: fix integer-overflow issues in main program * NEWS: Document this. * bootstrap.conf (gnulib_modules): Add inttypes, xstrtoimax. Remove xstrtoumax. * src/main.c: Include , for INTMAX_MAX, PRIdMAX. (context_length_arg, prtext, grepbuf, grep, grepfile) (get_nondigit_option, main): Use intmax_t, not int, for line counts. (context_length_arg, main): Silently ceiling line counts to maximum value, since there's no practical difference between doing that and using infinite-precision arithmetic. (out_before, out_after, pending): Now intmax_t, not int. (max_count, outleft): Now intmax_t, not off_t. (prepend_args, prepend_default_options, main): Use size_t, not int, for sizes. (prepend_default_options): Check for int and size_t overflow. grep: avoid mishandling of long lines * src/pcresearch.c (Pexecute): Do not pass a line longer than INT_MAX to pcre_exec, since its API does not permit that. grep: remove no-longer-used setrlimit code This code has been unused and obsolescent ever since the regex code stopped using the stack for large regular expressions. * src/main.c [HAVE_SETRLIMIT]: Do not include or or ; no longer needed. (set_rlimits): Remove. All callers changed. grep: fix some core dumps with long lines etc. These problems mostly occur because the code attempts to stuff sizes into int or into unsigned int; this doesn't work on most 64-bit hosts and the errors can lead to core dumps. * NEWS: Document this. * src/dfa.c (token): Typedef to ptrdiff_t, since the enum's range could be as small as -128 .. 127 on practical hosts. (position.index): Now size_t, not unsigned int. (leaf_set.elems): Now size_t *, not unsigned int *. (dfa_state.hash, struct mb_char_classes.nchars, .nch_classes) (.nranges, .nequivs, .ncoll_elems, struct dfa.cindex, .calloc, .tindex) (.talloc, .depth, .nleaves, .nregexps, .nmultibyte_prop, .nmbcsets): (.mbcsets_alloc): Now size_t, not int. (dfa_state.first_end): Now token, not int. (state_num): New type. (struct mb_char_classes.cset): Now ptrdiff_t, not int. (struct dfa.utf8_anychar_classes): Now token[5], not int[5]. (struct dfa.sindex, .salloc, .tralloc): Now state_num, not int. (struct dfa.trans, .realtrans, .fails): Now state_num **, not int **. (struct dfa.newlines): Now state_num *, not int *. (prtok): Don't assume 'token' is no wider than int. (lexleft, parens, depth): Now size_t, not int. (charclass_index, nsubtoks) (parse_bracket_exp, addtok, copytoks, closure, insert, merge, delete) (state_index, epsclosure, state_separate_contexts) (dfaanalyze, dfastate, build_state, realloc_trans_if_necessary) (transit_state_singlebyte, match_anychar, match_mb_charset) (check_matching_with_multibyte_ops, transit_state_consume_1char) (transit_state, dfaexec, free_mbdata, dfaoptimize, dfafree) (freelist, enlist, addlists, inboth, dfamust): Don't assume indexes fit in 'int'. (lex): Avoid overflow in string-to-{hi,lo} conversions. (dfaanalyze): Redo indexing so that it works with size_t values, which cannot go negative. * src/dfa.h (dfaexec): Count argument is now size_t *, not int *. (dfastate): State numbers are now ptrdiff_t, not int. * src/dfasearch.c: Include "intprops.h", for TYPE_MAXIMUM. (kwset_exact_matches): Now size_t, not int. (EGexecute): Don't assume indexes fit in 'int'. Check for overflow before converting a ptrdiff_t to a regoff_t, as regoff_t is narrower than ptrdiff_t in 64-bit glibc (contra POSIX). Check for memory exhaustion in re_search rather than treating it merely as failure to match; use xalloc_die () to report any error. * src/kwset.c (struct trie.accepting): Now size_t, not unsigned int. (struct kwset.words): Now ptrdiff_t, not int. * src/kwset.h (struct kwsmatch.index): Now size_t, not int. tests: test for problems with long matches The new test is expensive, so add a category of expensive tests, which are normally not run, and put the new test in this new category. The idea of having expensive tests is taken from coreutils. * HACKING: Mention RUN_EXPENSIVE_TESTS and similar env vars. * Makefile.am (check-expensive): New rule. * tests/Makefile.am (TESTS): Add big-match. * tests/init.cfg (expensive_): New function, from coreutils. * tests/big-match: New file. 2012-02-29 Paul Eggert maint: use gnulib _Noreturn rather than __attribute__ ((noreturn)) * src/grep.h (__attribute__): Remove. * src/dfa.h (__attribute__): Likewise. (dfaerror): Use noreturn rather than __attribute__ ((noreturn)). * src/main.c (usage): Likewise. 2012-02-26 Jim Meyering build: update submodule, bootstrap, tests/init.sh from gnulib * gl/lib/regcomp.c.diff: Adjust. * bootstrap: Update from gnulib. * tests/init.sh: Update from gnulib. 2012-02-26 Paolo Bonzini dfa: merge calls to SUCCEEDS_IN_CONTEXT * src/dfa.c (state_index): use a single call to SUCCEEDS_IN_CONTEXT. dfa: fix a subtle constraint encoding bug * src/dfa.c (SUCCEEDS_IN_CONTEXT, PREV_NEWLINE_DEPENDENT, PREV_LETTER_DEPENDENT): Rewrite to handle all 3*3=9 possible combinations of previous and next character contexts. (MATCHES_NEWLINE_CONTEXT, MATCHES_LETTER_CONTEXT): Remove. (NO_CONSTRAINT, BEGLINE_CONSTRAINT, ENDLINE_CONSTRAINT, BEGWORD_CONSTRAINT, ENDWORD_CONSTRAINT, LIMWORD_CONSTRAINT, NOTLIMWORD_CONSTRAINT): Switch to new encoding. * NEWS: Document resulting bugfix. * tests/spencer1.tests: Add regression test. dfa: do not use MATCHES_*_CONTEXT directly * src/dfa.c (dfastate): Use SUCCEEDS_IN_CONTEXT. dfa: change meaning of a state context * src/dfa.c (MATCHES_NEWLINE_CONTEXT, MATCHES_LETTER_CONTEXT): New. (state_separate_contexts): Remove second argument. (state_index): Do not mask away CTX_NONE. (dfaanalyze): Adjust call to state_index and state_separate_contexts. (dfastate): Adjust calls to state_index and state_separate_contexts. 2012-02-13 Paul Eggert tests: fix loop in epipe test * tests/epipe: Don't loop forever if the bug is present. Problem reported by Jaroslav Skarvada. 2012-02-08 Paul Eggert tests: work portably even if SIGPIPE is ignored * tests/epipe: Don't rely on "trap - PIPE"; that's not portable. Problem reported by Eric Blake in . Also, use "ls -al" rather than "echo", in case "echo" is done by a buggy shell that ignores write errors. And close grep's fd 3, as a sanity check. 2012-02-07 Paul Eggert tests: work even if SIGPIPE is ignored * tests/epipe: Do not infinite-loop if SIGPIPE is already ignored. It could be that the invoker of 'make check' ignores SIGPIPE, for example. 2012-02-05 Jim Meyering build: accommodate -Wshadow and -Werror=suggest-attribute=pure * src/dfa.c (state_separate_contexts): Add _GL_ATTRIBUTE_PURE. (dfaexec): Rename parameter, s/newline/allow_nl/, to avoid shadowing the global. 2012-02-05 Paolo Bonzini dfa: refactor common context computations * src/dfa.c (CTX_ANY, charclass_context, state_separate_contexts): New. (dfaanalyze): Use state_separate_contexts. (dfastate): Use charclass_context and state_separate_contexts. Rename prev_context to separate_contexts. dfa: change newline/letter to a single context value * src/dfa.c (MATCHES_NEWLINE_CONTEXT, MATCHES_LETTER_CONTEXT, SUCCEEDS_IN_CONTEXT, ACCEPTS_IN_CONTEXT): Take a single context value for prev and curr. (struct dfa_state): Replace newline and letter with context. (wchar_context): New. (state_index): Replace newline and letter with context. Compare context values in the state struct. Adjust calls to pass contexts. (wants_newline): Replace with wanted_context. Adjust calls to pass contexts. (dfastate): Replace wants_newline and wants_letter with wanted_context. Adjust calls to pass contexts. (build_state): Adjust calls to pass contexts. (match_anychar, match_mb_charset, transit_state): Use wchar_context. Adjust calls to pass contexts. 2012-02-05 Paolo Bonzini dfa: introduce contexts for the values in d->success Also initialize all tables in a single place in dfasyntax. * src/dfa.c (CTX_NONE, CTX_LETTER, CTX_NEWLINE, char_context): New. (sbit, letters, newline): New. (dfasyntax): Fill them. (dfastate): Remove letters, newline, initialized. (build_state): Use CTX_* constants. (dfaexec): Remove sbit and sbit_init. 2012-02-05 Paolo Bonzini dfa: remove useless check * src/dfa.c (state_index): There is nothing that is a newline *and* a letter. Remove redundant call to SUCCEEDS_IN_CONTEXT. 2012-01-22 Jim Meyering build: update bootstrap from gnulib and adapt * bootstrap: Update from gnulib. * tests/init.sh: Update from gnulib. * bootstrap.conf (bootstrap_epilogue): Remove now-unnecessary, snippet that edited gnulib-tests/gnulib.mk. (gnulib_tool_option_extras): Add both --symlink and --makefile-name=gnulib.mk. Remove use of $bt. * lib/Makefile.am: Initialize numerous automake variables so that generated code in gnulib.mk may use += to append to them. maint: convert `this' to 'this' quoting style in diagnostics Now that gnulib's quote and quotearg modules use 'this' style, change the few explicit uses in diagnostics to conform. * src/egrep.c (after_options): Use 'this' style of quotes. * src/fgrep.c (after_options): Likewise. * src/grep.c (after_options): Likewise. * src/main.c (usage): Likewise. build: update gnulib to latest; adjust quoting in tests * gnulib: Update. * tests/in-eq-out-infloop: Convert expected diagnostics to match new quoting. 2012-01-22 Paul Eggert doc: document recent diagnostics-related changes * NEWS: Document changes re diagnostics related to GREP_COLORS, directory loops, -s, "write error". grep: be quiet about GREP_COLORS syntax * src/main.c (struct color_cap): fct now returns void, since there's no longer need to use what it returns. (color_cap_mt_fct, color_cap_rv_fct, color_cap_ne_fct): Return void. (parse_grep_colors): Do not output diagnostics and then exit with status 0. Instead, ignore errors in GREP_COLORS. This is more consistent with programs that (e.g.) ignore errors in termcap entries, and it's more internally-consistent as some GREP_COLORS errors were ignored but not others. grep: exit with nonzero status if directory loop * src/main.c (grepdir): Exit with status 2 if a directory loop is found, since the output might not be "right" (i.e., infinite...). grep: suppress read errors if -s * src/main.c (reset, grep, grepfile): Do not report an input error if -s is given. grep: don't say "write error" over and over Problem reported by Travis Gummels in . * src/main.c (write_error_seen): New static var. (clean_up_stdout): New function. (prline): Do not output 'write error' more than once; exit after the first one. Use the same wording for the diagnostic that close_stdout uses. (main): Clean up with clean_up_stdout, not close_stdout, so that grep doesn't output multiple "write error" diagnostics. * tests/Makefile.am (TESTS): Add epipe. * tests/epipe: New file. 2012-01-12 Paul Eggert dfa: non-glibc word-constituent unibyte fix * src/dfa.c (is_valid_unibyte_character): Fix typo that caused this to incorrectly return 0 on unibyte non-glibc systems. Problem reported by Aharon Robbins in . 2012-01-04 Paul Eggert doc: document empty pattern better * doc/grep.texi (Top, Fundamental Structure, Usage): Explain how grep deals with the empty pattern. Problem spotted by Bernhard Voelker in . grep: with no args, search "." only if command-line -r * NEWS: Document this. * doc/grep.texi (Environment Variables, grep Programs): Likewise. * src/main.c (usage): Likewise. (main): Implement this. (prepend_default_options): Return a count of prepended options. * tests/r-dot: Test the above. 2012-01-03 Jim Meyering tests: adjust test to match code, now that --mmap writes to stderr * tests/ignore-mmap: Separate stdout and stderr; test both. deprecate the --mmap option * src/main.c (main): Deprecate the --mmap option: issue a warning when it is used. (usage): Change description. * doc/grep.texi (Other Options): Document the new behavior. * NEWS (Changes in behavior): Mention it. 2012-01-03 Paolo Bonzini dfa: fix incorrect comment * src/dfa.c (dfastate): Fix comment for newline. dfa: fix rebase conflict * src/dfa.c (dfaanalyze): Fix reference to nalloc. dfa: automatically resize position_sets * src/dfa.c (insert, copy, merge): Resize arrays here. (dfaanalyze): Do not track number of allocated elements here. (dfastate): Allocate mbps with only one element. dfa: change position_set nelem to size_t * src/dfa.c (REALLOC_IF_NECESSARY): Disable assertion, to avoid warnings from -Wtype-limits. (position_set): Change nelem to a size_t. dfa: move nalloc to position_set structure * src/dfa.c (position_set): Add alloc. (alloc_position_set): Initialize it. (dfaanalyze): Use it instead of the nalloc array or nelem. dfa: remove dead assignment * src/dfa.c (transit_state): transit_state_consume_1char will clear follows, do not do this ourselves. dfa: introduce alloc_position_set * src/dfa.c (alloc_position_set): New function, use it throughout. dfa: use a more compact data type for grps * src/dfa.c (leaf_set): New. (dfastate): Use the smaller type, leaf_set, for grps. Its prior type contained an unused constraint field. dfa: use MALLOC/REALLOC always src/dfa.c (dfastate, enlist, dfamust): Use MALLOC and REALLOC. dfa: remove unnecessary braces * src/dfa.c (dfastate): Remove unnecessary braces. dfa: x2nrealloc starting from a NULL pointer works * src/dfa.c (parse_bracket_exp): Do not MALLOC mbcset parts the first time they are encountered. Initialize chars_al correctly. 2012-01-03 Jim Meyering build: avoid build failure with --enable-gcc-warnings and recent gcc * lib/colorize-posix.c: Disable -Wsuggest-attribute=const, to avoid warning about this empty init_colorize function. 2012-01-03 Paolo Bonzini remove lib/ms/ * configure.ac: Create lib/colorize.c as a symbolic link. * lib/colorize-posix.c: New name of lib/colorize-impl.c. * lib/colorize-w32.c: New name of lib/ms/colorize-impl.c. * lib/colorize.c: Delete. * lib/Makefile.am (EXTRA_DIST): Adjust. * .gitignore: Adjust. * cfg.mk: Adjust syntax-check exclusions. unify colorize.h headers * lib/Makefile.am (EXTRA_DIST): Adjust. * lib/colorize.h: Remove inline functions. * lib/colorize-impl.c: Move them here as functions. * lib/ms/colorize.h: Remove. * src/Makefile.am (DEFAULT_HEADERS): Remove. 2012-01-02 Paolo Bonzini colorize: use isatty module * bootstrap.conf: Add isatty module. * gnulib: Update to latest. * lib/colorize.h: Remove argument from should_colorize. * lib/ms/colorize.h: Likewise. * lib/colorize-impl.c: Factor isatty call out of here... * lib/ms/colorize-impl.c: ... and here... * src/main.c: ... into here. 2012-01-02 Jim Meyering tests: avoid minor "make check" failure * tests/r-dot: Make executable, to avoid triggering a failed consistency test in "make check". 2012-01-02 Paul Eggert grep: -r with no args now searches "." This is a patch I've been meaning to put in for years. When I added support for "grep -r", I forgot to have "grep -r PAT" search the working directory by default, instead of searching standard input (which makes no sense, even if stdin is a directory). This is not an upward compatible change, since "grep -r PAT grep: check stdin like other files * NEWS: Document this. * src/main.c (grepfile): Revamp tests for input files so that standard input is tested like other files. For example, report an error if standard input equals standard output. Prefer open+fstat to stat+open if possible, as open+fstat is usually a bit faster and avoids a race condition. * tests/in-eq-out-infloop: Add tests for cases like 'grep pat >file'. 2012-01-01 Jim Meyering maint: update all copyright year number ranges Run "make update-copyright". 2011-12-31 Paul Eggert grep: lower-case function names These names used to be macros, but they're functions now. All callers changed. * src/main.c (pr_sgr_start): Rename from PR_SGR_START. (pr_sgr_end): Rename from PR_SGR_END. (pr_sgr_start_if): Rename from PR_SGR_START_IF. (pr_sgr_end_if): Rename from PR_SGR_END_IF. ms: move Microsoft-specific stuff to lib/ms * cfg.mk (exclude_file_name_regexp--sc_prohibit_strcmp) (exclude_file_name_regexp--sc_require_config_h) (exclude_file_name_regexp--sc_require_config_h_first): New rules. * lib/colorize.c, lib/colorize.h, lib/colorize-impl.c: * lib/ms/colorize.h, lib/ms/colorize-impl.c: New files. * configure.ac (GREP_SRC_INCLUDES): New macro. * lib/Makefile.am (libgreputils_a_SOURCES): Add colorize.[ch]. (EXTRA_DIST): New macro. * src/Makefile.am (DEFAULT_INCLUDES): New macro. * src/main.c: Include colorize.h. (PR_SGR_START, PR_SGR_END, PR_SGR_START_IF, PR_SGR_END_IF): Now static functions, not macros. (hstdout, norm_attr, w32_console_init, w32_sgr2attr) (w32_clreol) [__MINGW32__]: Move to lib/ms/colorize-impl.c. (pr_sgr_start, pr_sgr_end): Remove; callers changed to use new print_start_colorize, print_end_colorize from colorize.h. (init_colorize): Rename from w32_console_init and move to colorize module; caller changed. (should_colorize): Move to colorize module. grep: do input==output check more like dir loop check * src/main.c (grepfile): Just use SAME_INODE; don't bother with SAME_REGULAR_FILE. This works better on properly-working POSIX hosts, since it handles the case where the file is changing as we grep it. It works worse on hosts that don't support st_ino properly, but in practice this isn't that much of a problem here. * src/system.h (same_file_attributes, SAME_REGULAR_FILE): Remove; no longer needed. build: update gnulib submodule to latest 2011-12-28 Paul Eggert maint: remove now-unused/obsolete files * README.DOS: Remove file. * m4/djgpp.m4: Likewise. * .gitignore: Remove reference to m4/djgpp.m4. 2011-12-28 Jim Meyering maint: distribute ChangeLog-2009 * Makefile.am (EXTRA_DIST): Add ChangeLog-2009. Spotted by Eli Zaretskii. 2011-12-28 Jim Meyering main.c: add some 'const' directives * src/main.c (color_dict, fg_color, bg_color, cap): Declare const. No semantic change. 2011-12-28 Jim Meyering main.c: correct indentation and formatting style * src/main.c: Correct many formatting inconsistencies. No semantic change. avoid new syntax-check failures * cfg.mk (old_NEWS_hash): Update, to accommodate old NEWS modification. * src/main.c: Indent solely with spaces, never with TABs. (should_colorize): Remove useless parens in #if directive. 2011-12-28 Eli Zaretskii Fix whitespace, indentation and documentation * src/main.c (parse_grep_colors): Fix indentation. (usage): Mention MS-Windows in help text for -U and -u options. update NEWS for MS-Windows changes * NEWS: Mention MS-Windows related bugfixes and enhancements. Fix the test suite for MS-Windows. * tests/include-exclude: Use --directories=skip, to avoid gratuitous failures on systems that cannot grep directories. * tests/reversed-range-endpoints: Don't reject program names with leading directories and drive letters. * tests/warn-char-classes: Likewise. Support color highlighting on MS-Windows * src/main.c (SGR_START, SGR_END, PR_SGR_FMT, PR_SGR_FMT_IF): Remove. (PR_SGR_START, PR_SGR_START_IF): Replace with pr_sgr_start. (PR_SGR_END, PR_SGR_END_IF): Replace with pr_sgr_end. (pr_sgr_start, pr_sgr_end, should_colorize): New functions. (w32_console_init, w32_sgr2attr, w32_clreol) [__MINGW32__]: New functions. (main): Use should_colorize. Invoke w32_console_init. 2011-12-24 Paul Eggert don't ignore errors when reading a directory grep no longer silently suppresses errors when reading a directory as if it were a text file. For example, "grep x ." now reports a read error on most systems; formerly, it ignored the error. Problem reported as an aside by Bob Proulx (Bug#10355). * NEWS: Document this. * src/main.c (grep, grepfile): Implement this. Simplify the code considerably. * src/system.h (is_EISDIR): Remove; no longer needed. --include etc. now work on command-line args more consistently --include and --exclude apply only to non-directories and --exclude-dir applies only to directories. "-" (standard input) is never excluded, since it is not a file name. This bug was discovered while fixing a read-directory bug (Bug#10355). * NEWS: Document this. * src/main.c (main): Implement this. * tests/include-exclude: Test for it. 2011-12-24 Jim Meyering build: update gnulib submodule to latest 2011-12-12 Arnold D. Robbins doc: improve grep.texi * doc/grep.texi: General editing for improved aesthetics. Also fix a few problems. 2011-12-12 Jim Meyering build: use gnulib's iswctype wcscoll * bootstrap.conf (gnulib_modules): Add iswctype and wcscoll. * configure.ac: Remove explicit checks for those functions. * src/mbsupport.h (MBS_SUPPORT): Define to 1 if not already defined. Remove the conditional, now that we're guaranteed by gnulib to have wcscoll and iswctype. Suggested by Alan Hourihane in http://savannah.gnu.org/bugs/?34930 disable the new input==output guard for additional options * src/main.c (grepfile): Do not reject input == output also when using a few other options. * tests/in-eq-out-infloop: Test these new cases. * NEWS (Bug fixes): Mention it 2011-12-11 Nicolas Vigier do not reject "grep -qr . > out" The recent fix to avoid an infinite disk-filling loop, commit 5e20a38a, introduced a minor regression. If you use grep with -q and -r, and redirect output to a file that will be traversed, then grep would reject the command, even though it will generate no output. In that case, there is no risk of an infinite loop. * src/main.c (grepfile): Do not reject input == output when using --quiet/--silent (-q). Reported by J H Wilson in http://bugs.mageia.org/show_bug.cgi?id=3501 forwarded by Nicolas Vigier to https://savannah.gnu.org/bugs/?34917 2011-11-29 Arnold Robbins dfa: do not call nl_langinfo in !MBS_SUPPORT mode * src/dfa.c (using_utf8) [!MBS_SUPPORT]: Remove erroneous "defined" in cpp test for MBS_SUPPORT. Since commit a163349d, MBS_SUPPORT is 0/1. This error caused trouble only in the !MBS_SUPPORT case. dfa: avoid warning from deficient compiler in !MBS_SUPPORT mode * src/dfa.c (setbit_wc) [!MBS_SUPPORT]: Add explicit "return false;" after "abort ();", to avoid a warning from deficient compilers. 2011-11-29 Jim Meyering tests: use "compare exp out", not "compare out exp" Likewise, when an empty file is expected, use "compare /dev/null out", not "compare out /dev/null". I.e., specify the expected/desired contents via the first file name. Prompted by a suggestion from Bruno Haible in http://thread.gmane.org/gmane.comp.gnu.grep.bugs/4020/focus=29154 Run these commands: git grep -l -E 'compare [^ ]+ exp' \ |xargs perl -pi -e 's/(compare) (\S+) (exp\S*)/$1 $3 $2/' git grep -l -E 'compare [^ ]+ /dev/null' \ |xargs perl -pi -e 's/(compare) (\S+) (\/dev\/null)/$1 $3 $2/' 2011-11-29 Jim Meyering build: update gnulib submodule to latest 2011-11-28 Jim Meyering build: accommodate -Werror=suggest-attribute=pure Now that we're using the latest manywarnings module from gnulib, accommodate gcc's -Werror=suggest-attribute=pure option by marking suggested functions with gnulib-defined _GL_ATTRIBUTE_PURE. * src/kwset.c (hasevery): Mark function with pure attribute. (bmexec): Likewise. * src/dfa.c (nsubtoks, istrstr, find_pred, dfamusts): Likewise. * configure.ac: Disable (for lib/) options that seem not to be worth the trouble: -Wunsuffixed-float-constants and -Wformat-nonliteral. 2011-11-21 Bruno Haible build: fix "make check" error on OSF/1 * tests/Makefile.am (TESTS_ENVIRONMENT): Test the value of the variable BASH_VERSION, not the literal ASH_VERSION. 2011-11-21 Jim Meyering portability: work consistently on *BSD systems * src/dfa.c (is_valid_unibyte_character): Define. (IS_WORD_CONSTITUENT): Use it here, to make grep work consistently even on *BSD systems, which use different tables for ctype macros like isalpha. http://thread.gmane.org/gmane.comp.gnu.grep.bugs/4022 With help from Bruno Haible. 2011-11-20 Jim Meyering maint: consistently use NULL, not 0, when comparing pointers * src/dfa.c (dfaanalyze): Compare trans[s] with NULL, not 0. maint: remove an avoidable #ifdef/#endif pair * src/dfa.c (dfaanalyze): Remove avoidable #ifdef around "{". tests: fix typo in last change * tests/word-delim-multibyte: Use double quotes around $e_acute, not single quotes. Spotted by Bruno Haible. This and the preceding change do not resolve the XPASS failure on OpenBSD 4.9 after all. See the explanation at http://thread.gmane.org/gmane.comp.gnu.grep.bugs/4022 tests: avoid unwarranted test failure on *BSD-based systems * tests/word-delim-multibyte (e_acute): Use a more portable representation of e-acute. Reported by Bruno Haible. 2011-11-19 Jim Meyering maint: accommodate -Wdeclaration-after-statement, but only in dfa.c, and because doing so does not impact readability/maintainability. This is solely to accommodate gawk users who are stuck with ancient gcc. This is no excuse to change any other code in grep. * src/dfa.c (dfaoptimize, parse_bracket_exp): Move declaration to precede first statement in block. 2011-11-16 Jim Meyering maint: post-release administrivia * NEWS: Add header line for next release. * .prev-version: Record previous version. * cfg.mk (old_NEWS_hash): Auto-update. version 2.10 * NEWS: Record release date. build: update gnulib submodule to latest 2011-11-13 Jim Meyering maint: update bootstrap and init.sh from gnulib * tests/init.sh: Update from gnulib. * bootstrap: Likewise. 2011-11-12 Jim Meyering build: update gnulib for exclude-test fixes tests: make our "export" replacement efficient with modern shells * tests/Makefile.am (TESTS_ENVIRONMENT): Use a trivial and efficient implementation with a shell that supports "export var=val". Use the sed-invoking replacement only when necessary. Improved by Stefano Lattarini. tests: make the replacement export function more robust * tests/Makefile.am (sed_quote_value): Also quote single quotes. Remove sed's -e options. Not needed. 2011-11-12 Bruno Haible tests: fix test suite execution failure on OSF/1 5.1 * tests/Makefile.am (TESTS_ENVIRONMENT): Use a shell function to ensure that we use only the portable form of the 'export' shell built-in. tests: don't assume that /bin/bash exists * tests/fedora: Run using /bin/sh, not /bin/bash. tests: avoid unwarranted failures due to SATAN's timeout * tests/init.cfg (require_timeout_): Also ensure that timeout exits with its child's exit status. build: fix compilation error on MSVC 9 to due Pexecute() declaration * src/pcresearch.c (WITHOUT_PCRE_NORETURN): Remove macro. (Pexecute): Replace abort() call with code that does not trigger GCC warnings. tests: fix high-bit-range test failure on OSF/1 5.1 * tests/high-bit-range: Use octal escape instead of hexadecimal escape sequence. 2011-11-11 Jim Meyering build: update gnulib for solaris test fix 2011-11-10 Jim Meyering build: update gnulib submodule to latest maint: adjust the URL that will appear in the generated announcement * cfg.mk (url_dir_list): Use this http://ftp.gnu.org/gnu/$(PACKAGE) for the first link listed in the generated announcement. announce-gen now provides the faster mirror link automatically. 2011-11-06 Jim Meyering build: stop distributing gzip'd releases; xz is enough * configure.ac (AM_INIT_AUTOMAKE): Add no-dist-gzip. * NEWS (Build-related): Mention that we're dropping .tar.gz. build: update gnulib submodule to latest 2011-10-14 Stefano Lattarini distcheck: ensure dist-hook fails if syntax-check fails * Makefile.am (run-syntax-check): Fix logic, to ensure that the recipe of this target returns a non-zero exit status if "make syntax-check" fails. 2011-10-12 Jim Meyering build: update gnulib submodule to latest This should fix a few portability problems, including one on HP-UX and a test-float failure on PPC, reported by Andreas Metzler. 2011-10-10 Stefano Lattarini gitignore: merge top-level and tests/ .gitignore files * tests/.gitignore: Remove; what little remained of its contents has been moved ... * .gitignore: ... here. tests: tiny simplification in TESTS_ENVIRONMENT definition * tests/Makefile.am (TESTS_ENVIRONMENT): Remove redundant use of `export'. 2011-10-10 Stefano Lattarini tests: support development version of automake too This change implements a more correct and idiomatic use of the features of the Automake-provided 'parallel-tests' harness. Moreover, this change is required in order for the testsuite to continue to work with the new testsuite harness that is planned to be introduced in Automake 1.12 (which, as of the writing date, is still under development and in late alpha state). * tests/Makefile.am (TESTS_ENVIRONMENT): The development version of automake dos not support setting the interpreter delegated to run the tests scripts in this variable; instead, use ... (LOG_COMPILER): ... this variable. * .gitignore: Ignore `.trs' files in directory `tests/'. * build-aux/.gitignore: Ignore `test-driver' script. 2011-10-03 Eli Zaretskii dfa: don't mishandle high-bit bytes in a regexp with signed-char This appears to arise only on systems for which "char" is signed. * src/dfa.c (FETCH_WC, FETCH): Produce an unsigned value, rather than a sign-extended one. Fixes a bug on MS-Windows with compiling patterns that include characters with the 8-th bit set. (to_uchar): Define. From coreutils. Reported by David Millis . See http://thread.gmane.org/gmane.comp.gnu.grep.bugs/3893 * NEWS (Bug fixes): Mention it. 2011-09-16 Jim Meyering maint: dfa: simplify multi-byte-related conditionals * src/dfa.c (setbit_case_fold_c, parse_bracket_exp, lex): (addtok_mb, dfaparse): Change each "MBS_SUPPORT && MB_CUR_MAX > 1" test to just "MB_CUR_MAX > 1". * src/dfasearch.c (kwsincr_case, EGexecute): Likewise. * src/kwsearch.c (Fcompile, Fexecute): Likewise. * src/searchutils.c (kwsinit): Likewise. * src/dfa.c (parse_bracket_exp): Convert "if (!MBS_SUPPORT || MB_CUR_MAX == 1)" to "if (MB_CUR_MAX == 1)" and do this: - assert(!MBS_SUPPORT || MB_CUR_MAX == 1); + assert(MB_CUR_MAX == 1); maint: dfa: simplify several expressions * src/dfa.c (dfainit): Set d->mb_cur_max unconditionally, now that MB_CUR_MAX is always usable. With that, simplify all "MBS_SUPPORT && d->mb_cur_max > 1" to simply "d->mb_cur_max > 1". (dfastate, dfaexec, dfainit, dfafree): Simplify, removing each now-unnecessary "MBS_SUPPORT &&". maint: dfa: avoid in-function "#if MBS_SUPPORT" tests * src/dfa.c (setbit_case_fold_c): Remove "#if MBS_SUPPORT" in favor of simple "if (MBS_SUPPORT ...". (dfaexec, addtok): Likewise. maint: ensure that MB_CUR_MAX is defined even when !MBS_SUPPORT * src/mbsupport.h [!MBS_SUPPORT] (MB_CUR_MAX): Define to 1. build: fix compilation failure when MBS_SUPPORT is 0 * src/dfa.c (add_utf8_anychar): Always compile this function, but when MBS_SUPPORT is 0, give it an empty body. (prepare_wc_buf): Likewise. [! MBS_SUPPORT] (setbit_wc): Define to always abort. maint: dfa: simplify dfaoptimize * src/dfa.c (dfaoptimize): Simplify. (dfacomp): Remove now-redundant "if (MBS_SUPPORT)" guard, since dfaoptimize does nothing if !MBS_SUPPORT. maint: dfa: remove some #if MBS_SUPPORT guards * src/dfa.c: Replace a few "#if MBS_SUPPORT" directives with "if (MBS_SUPPORT)". Remove some altogether. maint: dfa: convert #if-MBS_SUPPORT (dfastate) * src/dfa.c (dfastate): Use regular "if", not #if MBS_SUPPORT. maint: dfa: convert #if-MBS_SUPPORT (dfastate) * src/dfa.c (dfastate): Use regular "if", not #if MBS_SUPPORT. maint: dfa: convert #if-MBS_SUPPORT (state_index) * src/dfa.c (state_index): Use regular "if", not #if MBS_SUPPORT. maint: dfa: convert #if-MBS_SUPPORT (dfaparse) * src/dfa.c (dfaparse): Use regular "if", not #if MBS_SUPPORT.' maint: dfa: convert #if-MBS_SUPPORT (copytoks) * src/dfa.c (copytoks): Use regular "if", not #if MBS_SUPPORT.' maint: dfa: convert #if-MBS_SUPPORT (lex) * src/dfa.c (lex): Use regular "if", not #if MBS_SUPPORT.' maint: dfa: convert #if-MBS_SUPPORT (parse_bracket_exp) * src/dfa.c (parse_bracket_exp): Use regular "if", not #if MBS_SUPPORT. maint: dfa: convert #if-MBS_SUPPORT (parse_bracket_exp) * src/dfa.c (parse_bracket_exp): Use regular "if", not #if MBS_SUPPORT. maint: dfa: convert #if-MBS_SUPPORT (parse_bracket_exp) * src/dfa.c (parse_bracket_exp): Use regular "if", not #if MBS_SUPPORT. maint: dfa: convert #if-MBS_SUPPORT (dfaexec) * src/dfa.c (dfaexec): Use regular "if", not #if MBS_SUPPORT. maint: dfa: convert #if-MBS_SUPPORT (dfaexec) * src/dfa.c (dfaexec): Use regular "if", not #if MBS_SUPPORT. Also add curly braces around multi-line if/else blocks. maint: dfa: remove #if-MBS_SUPPORT (free_mbdata) * src/dfa.c (free_mbdata): Remove the #if guard altogether. maint: dfa: convert #if-MBS_SUPPORT (dfaoptimize, dfacomp) * src/dfa.c (dfaoptimize, dfacomp): Use regular "if", not #if MBS_SUPPORT. maint: dfa: convert #if-MBS_SUPPORT (dfafree) * src/dfa.c (dfafree): Use regular "if", not #if MBS_SUPPORT. maint: dfa: convert #if-MBS_SUPPORT (parse_bracket_exp, part1) * src/dfa.c (parse_bracket_exp): Remove in-function #if MBS_SUPPORT. maint: remove #if-MBS_SUPPORT declaration guards * src/search.h: Don't bother to #if-out declarations. maint: convert #if-MBS_SUPPORT (EGexecute) * src/dfasearch.c (EGexecute): Remove in-function #if MBS_SUPPORT. maint: convert #if-MBS_SUPPORT (kwsincr_case) * src/dfasearch.c (kwsincr_case): Remove in-function #if MBS_SUPPORT. Move decl's down. maint: convert #if-MBS_SUPPORT (Fcompile, etc.) * src/kwsearch.c (Fcompile, Fexecute): Remove in-function #if MBS_SUPPORT. (Fcompile): Rearrange some declarations. No semantic change. maint: convert #if-MBS_SUPPORT (kwsinit) * src/searchutils.c (kwsinit): Remove in-function #if MBS_SUPPORT. maint: dfa: remove case-guarding #if-MBS_SUPPORT * src/dfa.c [DEBUG] (prtok): Remove now-useless #if-MBS_SUPPORT. 2011-09-15 Jim Meyering maint: remove #if MBS_SUPPORT around member declaration * src/dfa.c (dfastate): Don't #ifdef-out "mbps" position_set member. maint: dfa: remove #if MBS_SUPPORT around struct definition * src/dfa.c (struct mb_char_classes): Don't #ifdef-out declarations. build: avoid compilation failure when building without PCRE support * src/pcresearch.c [!HAVE_LIBPCRE] (WITHOUT_PCRE_NORETURN): Define to _Noreturn, not obsoleted-by-gnulib _GL_ATTRIBUTE_NORETURN. Reported by Eric Blake. tests: stop using skip_test_; use skip_ instead * tests/init.cfg (skip_test_): Remove definition. Use the improved skip_ function from init.sh, now that it has the same feature. * tests/euc-mb: s/skip_test_/skip_/ * tests/sjis-mb: Likewise. * tests/fmbtest: Likewise. tests: skip tests that require MBS support * tests/init.cfg (require_compiled_in_MB_support): New function. * tests/char-class-multibyte: Use it here, since this test cannot succeed without MBS support. * tests/equiv-classes: Likewise. * tests/euc-mb: Likewise. * tests/fgrep-infloop: Likewise. * tests/init.cfg: Likewise. * tests/prefix-of-multibyte: Likewise. * tests/turkish-I: Likewise. * tests/sjis-mb: Likewise. tests: make fmbtest explain (to stderr, not log) why it is skipped * tests/fmbtest: Use skip_ and fail_ to give better diagnostics. maint: dfa: improve comments * src/dfa.c (match_mb_charset, match_anychar): Improve comments. 2011-09-14 Jim Meyering build: update gnulib submodule to newer maint: correct indentation * src/dfa.c (dfaexec): Reposition curly braces to match indentation style. Remove useless comment. maint: move declaration "down" to inner scope where it is used * src/dfa.c (dfaexec): Move decl of local down into scope where used. 2011-09-07 Jim Meyering doc: use "file name" consistently in grep's --help output * src/main.c (usage): Use "file name", not "filename" in descriptions of --with-filename (-H), --no-filename (-h) and --label=LABEL. Suggested by Sequoia McDowell. bug: requires ru_RU.KOI8-R". [bug introduced in grep-2.9] 2011-08-31 Matthew Burgess tests: remove debug code that would cp to /t * tests/unibyte-bracket-expr: Remove debug artifact introduced by 2011-06-02 commit de5f7000, "tests: exercise a uni-byte [...] bug: requires ru_RU.KOI8-R". [bug introduced in grep-2.9] 2011-08-20 Jim Meyering build: use largefile module and update to latest gnulib * configure.ac: Remove AC_SYS_LARGEFILE, subsumed by ... * bootstrap.conf (gnulib_modules): ...this. Use largefile module. * gnulib: Update to latest. maint: clean up and plug a leak-on-OOM * src/dfa.c (icatalloc): Clean up; use xrealloc in place of malloc and realloc; remove conditionals that are unnecessary, now that failed allocation results in exit. (enlist): Use xrealloc in place of realloc; remove conditional. (comsubs): Avoid leak upon failed enlist call. (dfamust): Use xmalloc in place of malloc. Remove conditionals, now that icpyalloc and icatalloc never return NULL. maint: use x2nrealloc, not xrealloc * src/main.c (main): Use x2nrealloc, not xrealloc 2011-07-24 Jim Meyering tests: add a test to trigger the bug * tests/Makefile.am (TESTS): Add it. * tests/in-eq-out-infloop: Exercise the bug/fix. exit 2 (rather than infloop) when an input file is also on stdout This avoids a potential "infinite" disk-filling loop. Reported in http://savannah.gnu.org/patch/?5316 and http://savannah.gnu.org/bugs/?17457. * src/main.c: Include "quote.h". (out_stat): New global. (grepfile): Compare each regular file's dev/ino/etc. with those from the file on stdout (if it too is regular). (main): Set out_stat, if stdout is a regular file. * src/system.h: Include "same-inode.h". (same_file_attributes): Define. From diffutils. (SAME_REGULAR_FILE): Define. * bootstrap.conf (gnulib_modules): Use quote, not quotearg. Use same-inode. * NEWS (Bug fixes): Mention it. 2011-07-15 Reuben Thomas doc: improve documentation of character classes in the man page * doc/grep.in.1: Reword documentation of character classes. 2011-07-12 Jim Meyering dfa: remove unnecessary inclusion of verify.h * src/dfa.c: Don't include "verify.h". dfa: simplify use of *ALLOC macros * src/dfa.c (XNMALLOC, XCALLOC): Redefine without outer cast-to-(t *). (CALLOC, MALLOC, REALLOC): Remove type "t" parameter and adjust callers. dfa: change semantics of REALLOC_IF_NECESSARY's 3rd parameter * src/dfa.c (REALLOC_IF_NECESSARY): Change meaning of 3rd param, from "maximum index" to 1 greater than that: the required number of *P-sized elements. Note that only some of the uses of REALLOC_IF_NECESSARY needed to be adjusted, the others had already required an extra element. dfa: rename REALLOC_IF_NECESSARY param/local for clarity * src/dfa.c (REALLOC_IF_NECESSARY): Rename nalloc and new_nalloc to n_alloc and new_n_alloc. dfa: prepare for a semantic change in REALLOC_IF_NECESSARY * src/dfa.c (REALLOC_IF_NECESSARY): Remove "t" (type) parameter. Use (*p) instead. Adjust all callers. dfa: add braces to REALLOC_IF_NECESSARY definition * src/dfa.c (REALLOC_IF_NECESSARY): Add curly braces; use TABs to right-indent. 2011-06-28 Paolo Bonzini doc: improve documentation of character classes * doc/grep.texi (Character classes): Mention explicitly when examples refer to the C locale, explain better the general meaning of character classes. 2011-06-28 Jim Meyering dfa: fix the root cause of the heap overrun dfa's "insert" function was supposed to be maintaining the position list sorted on *decreasing* index, but since the 2009-12-09 "Speed up insert" commit, 62458291, it was using code that assumed the data were sorted on *increasing* index. As such, sometimes it would no longer merge constraints (not finding a match) and would append entries that normally would have matched and been merged. Those erroneous append operations resulted in the heap overrun fixed by 2011-06-17 commit 0b91d692 by doubling the array size. * src/dfa.c (insert): Fix the comparison. (dfaanalyze): Now that that's fixed, revert commit 0b91d692, allocating space for only d->nleaves entries, not double that. As far as I can tell, this change has no effect other than decreased memory usage, although it may improve performance slightly, since the resulting list of positions is half as long as it used to be. 2011-06-28 Paolo Bonzini dfa: use memcpy to copy position_sets * src/dfa.c (copy): Use memcpy. dfa: use copyset to copy charclasses * src/dfa.c (add_utf8_anychar): Change memcpy to copyset. gnulib: Update Fixes mmap-anon.m4 conflict with fn_grep, reported by Rainer Orth. 2011-06-21 Jim Meyering maint: update bootstrap from gnulib * bootstrap: Update to latest, so it no longer inserts empty lines in .gitignore files. * .gitignore: Let bootstrap move "!..." lines to end of file. post-release administrivia * NEWS: Add header line for next release. * .prev-version: Record previous version. * cfg.mk (old_NEWS_hash): Auto-update. version 2.9 * NEWS: Record release date. build: avoid a warning when building with --disable-perl-regexp... and --enable-gcc-warnings. * src/pcresearch.c (WITHOUT_PCRE_NORETURN): Define. Remove the unreachable return statement. Reported by Eric Blake. tests: ensure that each test script is executable This adds a rule run at "make check" time to ensure that test scripts are consistently executable. This change is not required for "make check", but makes it easier for people to run scripts manually, but that is discouraged because doing so makes it easy to omit important variable settings that are normally provided via TESTS_ENVIRONMENT. This change also makes each of the existing TESTS executable. * tests/Makefile.am (check_executable_TESTS): New rule. (check): Depend on it. * tests/{all_scripts}: chmod 755. Prompted by a report from Eric Blake. maint: update bootstrap from gnulib * bootstrap: Update from gnulib. maint: update po/POTFILES.in * po/POTFILES.in: Remove dfasearch.c, now that it no longer contains a translatable diagnostic. tests: include-exclude: avoid false positive failure on FreeBSD * tests/include-exclude: Avoid false-positive failure due to matching "a" in a directory on FreeBSD, when searching a directory without "-r". Search for '^aaa$' rather than just 'a'. Adjust test inputs and expected output files accordingly. dfa: remove some useless casts * src/dfa.c (icatalloc): Change type of "old" parameter from "char const *" to "char *". Don't cast-away const on realloc argument. Remove now-unnecessary const-discarding cast. Don't (void)-cast strcpy result. * src/dosbuf.c (undossify_input): Remove anachronistic cast-to-"char *" of realloc argument. dfa: more heap-allocation-related overflow protection * src/dfa.c (enlist): Use xnrealloc, not realloc. Also, remove unnecessary cast-to-(char *). (dfamust): Use xnmalloc, not malloc. Before, this code would return upon malloc failure (xnmalloc exits upon failure), but later, via the *ALLOC macros, it could already exit, so this new potential exit point is nothing new. The same applies to enlist, since it is called only through dfamust. tests: update init.sh; simplify TESTS_ENVIRONMENT * tests/init.sh: Update from coreutils. * tests/Makefile.am (TESTS_ENVIRONMENT): Remove shell_or_perl_ function. Instead, just use $(SHELL), since grep has no test that starts with #!/usr/bin/perl. 2011-06-20 Jim Meyering build: update gnulib submodule to latest build: avoid configure/gnulib-related errors * bootstrap.conf: Remove now-unnecessary code to exclude gettext/intl-related m4 tests. 2011-06-19 Jim Meyering maint: tighten up superfluous code * src/main.c (parse_grep_colors): Use xstrdup in place of xmalloc, a useless test, strlen, and strcpy. 2011-06-19 Paul Eggert dfa: avoid possibility of overflow * src/dfa.c (REALLOC_IF_NECESSARY, CALLOC, MALLOC, REALLOC): Use functions from xalloc.h to avoid overflow. * src/dfasearch.c (GEAcompile): Use xnrealloc rather than realloc. * src/pcresearch.c (Pcompile): Use xnmalloc, not xmalloc. 2011-06-17 Jim Meyering build: update gnulib submodule to latest dfa: correct two uses of btowc * src/dfa.c (setbit_c, setbit_case_fold_c): Compare the btowc return value against WEOF, not EOF. Suggested by Eli Zaretskii. On a system like MinGW with unsigned wint_t, comparing a btowc return value against EOF (-1) would always be false. dfa: don't overrun a malloc'd buffer for certain regexps * src/dfa.c (dfaanalyze): Allocate space for twice as many positions as there are leaves. Before this change, for some regular expressions, DFA analysis would have inserted far more "positions" than dfa->nleaves (up to double). Reported by Raymond Russell in http://savannah.gnu.org/bugs/?33547 * tests/dfa-heap-overrun: Trigger the overrun. * tests/Makefile.am (TESTS): Add it. * NEWS (Bug fixes): Mention it. 2011-06-08 Jim Meyering tests: don't ignore sjis-mb test failure I made changes that caused grep to segfault during "make check" -- as seen in dmesg output -- yet no test failed(!), and there was no trace of the segfault in the logs. * tests/sjis-mb (test_grep_reject): Ensure that output is empty. Don't ignore test failure. 2011-06-07 Paolo Bonzini dfa: optimize wide characters in a bracket expression * src/dfa.c (addtok): Compile characters to an alternation. Handle the case when nothing else remains in the MBCSET. dfa: refactor to prepare for upcoming optimizations * src/dfa.c (parse_bracket_exp): Move optimization of MBCSET from here... (addtok): ... to here. 2011-06-07 Paolo Bonzini dfa: correct handling of single-byte character ranges This provides a better fix for the unibyte-bracket-expr and high-bit-range testcases, and fixes the latent bug tested by bogus-wctob. * src/dfa.c (setbit_case_fold): Remove, replace with... (setbit_wc, setbit_c, setbit_case_fold_c): ... these. (parse_bracket_exp): Use setbit_case_fold_c when iterating over single-byte sequences. Use setbit_wc for multi-byte character sets, and setbit_case_fold_c for single-byte character sets. (lex): Use setbit_case_fold_c for single-byte character sets. 2011-06-07 Paolo Bonzini tests: exercise latent bug in character ranges * tests/bogus-wctob: New. * Makefile.am (TESTS): Add it. 2011-06-07 Jim Meyering tests: exercise a uni-byte [...] bug: requires ru_RU.KOI8-R * tests/unibyte-bracket-expr: New file. * tests/Makefile.am (TESTS): Add it. * init.cfg (require_ru_RU_koi8_r): New function. fix the [...] bug also for relatively unusual uni-byte encodings * src/dfa.c (setbit_case_fold): Also handle uni-byte locales like the one mentioned in the original report: see 2011-05-07 commit d98338eb. Re-reported by Santiago Ruano Rincón. Note that most uni-byte locales are not affected. * NEWS (Bug fixes): Mention it. tests: use skip_test_, not skip_ Use skip_test_, not skip_. The former prints its message both to the log file and to FD 9 (redirected to tty via tests/Makefile.am), while skip_ prints only to stderr, which goes to the log file. * tests/init.cfg (skip_test_): New function. Use skip_test_ in place of skip_ everywhere. * tests/fmbtest: s/skip_/skip_test_/ * tests/sjis-mb: Likewise. * tests/euc-mb: Likewise. tests: fmbtest: factor * tests/fmbtest: Factor out locale-name duplication. tests: fix skip-inducing typo in fmbtest * tests/fmbtest: Fix locale name typo (s/cz_CZ/cs_CZ/) that would cause this test to be skipped every time. 2011-06-07 Paolo Bonzini gnulib: adjust included modules * bootstrap.conf (gnulib_modules): Drop strtoul, rename wctype to wctype-h. 2011-05-21 Jim Meyering grep -P: don't abort upon exceeding PCRE's backtracking limit * src/pcresearch.c (Pexecute): Handle PCRE_ERROR_MATCHLIMIT. * tests/Makefile.am (XFAIL_TESTS): Remove pcre-abort. * tests/pcre-abort: Expect failure, no output, and increase the length of the input string, in case the backtracking limit is ever raised. Adjust comment. * NEWS (Bug fixes): Mention it. tests: show how to make grep -P abort * tests/pcre-abort: New file. Minimal testcase by Paolo Bonzini, derived from a report by www.beaver@list.ru. * tests/Makefile.am (TESTS): Add it. (XFAIL_TESTS): Add it here, too, since this test always fails, for now. tests: fix oddities in pcre-z * tests/pcre-z: Redirect stderr inside $(), not outside. Remove double quotes around $REGEX (which is just 'a') within double-quoted "$(...)". Split a long line. tests: factor out a new require_pcre_ function * tests/init.cfg (require_pcre_): New function, factored out of... * tests/pcre-z: ...here. Use the function. * tests/pcre: Likewise. tests: clean up pcre * tests/pcre: Skip (don't pass) the test when PCRE support is disabled. Don't redirect so much to /dev/null, now that all test output goes to pcre.log. Remove unnecessary braces and diagnostic about failing test. 2011-05-13 Jim Meyering post-release administrivia * NEWS: Add header line for next release. * .prev-version: Record previous version. * cfg.mk (old_NEWS_hash): Auto-update. version 2.8 * NEWS: Record release date. build: update gnulib, for fixed getcwd test build: update gnulib submodule to latest maint: remove syntax-checking sc_tight_scope rule * src/Makefile.am (sc_tight_scope): Remove rule. Now it's provided via gnulib's maint.mk. * cfg.mk (sc_tight_scope): Likewise. 2011-05-08 Jim Meyering maint: use consistent declaration syntax * src/grep.h (matchers): Declare consistently, so the sc_tight_scope rule detects this as an extern-marked variable. 2011-05-07 Jim Meyering maint: use gnulib's new readme-release module * bootstrap.conf (gnulib_modules): Add readme-release. (bootstrap_epilogue): Add the recommended perl one-liner. * README-release: Remove file; it is now generated from gnulib. * .gitignore: Add it. * gnulib: Update submodule to latest. tests: exercise bug with 0x80..0xff in [...] * tests/high-bit-range: New test, inspired by an example in the report by Igor O. Ladygin: http://bugs.debian.org/624387, via Santiago Ruano Rincón's http://savannah.gnu.org/bugs/?33198 * tests/Makefile.am (TESTS): Add it. fix a bug whereby echo c|grep '[c]' would fail for any c in 0x80..0xff * src/dfa.c (setbit_case_fold) [MBS_SUPPORT]: Set the bit also when wctob returns EOF. * NEWS (Bug fixes): Mention it. 2011-05-02 Reuben Thomas doc: correct comment about mmap * doc/grep.texi (Other Options) [--mmap]: This option is now ignored, so using it can have no effect on performance. 2011-05-02 Arnold D. Robbins build: move add_utf8_anychar into MBS ifdef 2011-05-01 Arnold D. Robbins maint: remove GAWK ifndef; no longer needed 2011-05-01 Jim Meyering maint: remove now-unnecessary use of gnulib's strtol module * bootstrap.conf (gnulib_modules): Remove now-obsolete "strtol". 2011-04-29 Jim Meyering maint: tweak README-release * README-release: Add note to check the NixOS/Hydra autobuilder results. 2011-04-28 Jim Meyering build: update gnulib submodule to latest maint: add the tight_scope syntax-checking rule This ensures that the only externally scoped symbols are ones that are explicitly marked as "extern" or white-listed like "main". * src/Makefile.am (sc_tight_scope): New rule, copied from coreutils. * cfg.mk (sc_tight_scope): Define, to hook to it from the top level. maint: mark some function declarations as extern * src/search.h: Add "extern" keyword to each function declaration. 2011-04-23 Jim Meyering maint: fix doubled-word typos in comments * src/dfa.c (SUCCEEDS_IN_CONTEXT): Remove doubled "a". * src/dfa.c (BACKREF): s/it it/it is/ 2011-04-09 Jim Meyering maint: fix typos in comments: s/can not/cannot/ * src/dfa.c (check_matching_with_multibyte_ops, dfastate): As above. 2011-03-19 Jim Meyering maint: stop using .x-sc_* files to list syntax-check exemptions Instead, use the new mechanism with which you merely use a variable (derived from the rule name) defined in cfg.mk to an ERE matching the exempted file names. * gnulib: Update to latest, to get maint.mk that implements this. * .x-sc_bindtextdomain: Remove file. * .x-sc_prohibit_tab_based_indentation: Likewise. * .x-sc_prohibit_xalloc_without_use: Likewise. * .x-sc_space_tab: Likewise. * cfg.mk: Define variables to exempt the same files. build: correct my change of 2011-01-28 Do not override original dist-hook rule. * Makefile.am (run-syntax-check): Rename from overriding dist-hook. (dist-hook): Depend on run-syntax-check. 2011-02-27 Jim Meyering maint: update from gnulib * bootstrap: Update from gnulib. * tests/init.sh: Likewise. * gnulib: Update to latest. 2011-01-27 Jim Meyering build: update gnulib submodule to latest build: run syntax-check rules as part of "make dist" * Makefile.am (dist-hook): Depend on syntax-check. Suggested by Reuben Thomas. 2011-01-26 Jim Meyering maint: remove unneeded #include directives * lib/savedir.c: Don't include . Not needed. * src/dfa.c: Likewise. 2011-01-22 Jim Meyering build: avoid new syntax-check failures * .x-sc_bindtextdomain: New file, used to avoid a spurious failure from the new syntax-check rule. * NEWS: Remove a trailing space. 2011-01-19 Jim Meyering tests: add a known-to-fail test * tests/turkish-I: New test. * tests/Makefile.am (TESTS): Add it. (XFAIL_TESTS): Add here, too. Reported by Ilya Basin. maint: sort test names in Makefile.am * tests/Makefile.am (TESTS): Sort test names. 2011-01-05 Jim Meyering doc: remove erroneous "{,m}" item from grep man page * doc/grep.in.1: Remove item describing bogus {,m} regex notation. Reported by Fernando Basso. 2011-01-03 Jim Meyering maint: update copyright year ranges to include 2011 Run "make update-copyright", so "make syntax-check" works in 2011. build: update gnulib submodule to latest 2010-12-20 Paolo Bonzini main: fix exit status on xmalloc failures * NEWS: Update. * src/main.c (main): Set exit_failure. Reported by Guy Shaw. add comment above fn_grep * configure.ac (fn_grep): Add comment suggested by Bruno Haible. 2010-11-14 Paolo Bonzini grep: add include guards * src/system.h: Add multiple inclusion guards. * src/grep.h: Likewise. configure: fix M4 quotation * configure.ac: Add extra brackets around [...] patterns. configure: remove dependency on grep that supports long lines and -e * configure.ac (fn_grep): New. Set GREP and EGREP to it, replace with newly-built grep before AC_OUTPUT. Reported by Florin Iucha . 2010-11-04 Jim Meyering build: update gnulib to latest tests: don't hard-code a 5-second timeout; that's not always enough Instead, time the command in the C locale and use 10 times that duration -- rounded up to whole seconds -- as the timeout when running it in the UTF-8 locale. * tests/backref-multibyte-slow: Compute a performance-relative timeout. Reported by Gilles Espinasse, regarding an imac 400. For more details, see http://thread.gmane.org/gmane.comp.gnu.grep.bugs/3360 2010-10-09 Jim Meyering maint: describe policy on copyright year number ranges * README: Mention coreutils' long-standing policy on use of M-N ranges in copyright year lists. Requested by Richard Stallman. 2010-10-04 Dmitry V. Levin build: compile gnulib without -Wcast-align to avoid warnings on ARM * configure.ac (GNULIB_WARN_CFLAGS): Remove -Wcast-align. 2010-09-30 Jim Meyering maint: don't define a gpg_key_ID. now it's obtained automatically * cfg.mk (gpg_key_ID): Remove definition. No longer needed. 2010-09-23 Paolo Bonzini tests: add testcase for previous fix * tests/inconsistent-ranges: New. * tests/Makefile.am (TESTS): Add it. 2010-09-23 Paolo Bonzini dfa: process range expressions consistently with system regex The actual meaning of range expressions in glibc is not exactly strcoll, which makes the behavior of grep hard to predict when compiled with the system regex. Leave to the system regex matcher the decision of which single-byte characters are matched by a range expression. This partially reverts a change made in commit 0d38a8bb (which made sense at the time, but not now that src/dfa.c is not doing multibyte character set matching anymore). * src/dfa.c (in_coll_range): Remove. (parse_bracket_exp): Use system regex to find which single-char bytes match a range expression. 2010-09-23 Bruno Haible build: fix link error on systems that have libiconv but not libintl * src/Makefile.am (LDADD): Add $(LIBICONV). 2010-09-21 Jim Meyering build: avoid compilation failure on the Hurd * src/dfasearch.c (dfawarn): Rename enum symbols to use DW_ prefix, so as not to collide with "GNU", which is defined by the Hurd. Reported by Matthias Lanzinger in http://savannah.gnu.org/bugs/?31096 2010-09-20 Jim Meyering maint: avoid obsolete gnulib modules * bootstrap.conf (gnulib_modules): Don't use obsolete atexit module. Use malloc-gnu and realloc-gnu -- malloc and realloc are obsolete. maint: update README-release * README-release: Reflect changes in coreutils' version of this file. 2010-09-20 Aharon Robbins dfa: fix compilation when not using MBS * src/dfa.c (prepare_wc_buf) [!MBS_SUPPORT]: Do not compile this function. 2010-09-16 Jim Meyering post-release administrivia * NEWS: Add header line for next release. * .prev-version: Record previous version. * cfg.mk (old_NEWS_hash): Auto-update. version 2.7 * NEWS: Record release date. 2010-09-13 Paolo Bonzini tests: add equiv-classes * configure.ac (USE_INCLUDED_REGEX): Add Automake conditional. * tests/equiv-classes: New test. * tests/Makefile.am (TESTS): Add it. (XFAIL_TESTS) [USE_INCLUDED_REGEX]: Mark it as expected failure. 2010-09-13 Paolo Bonzini dfa: fall back to glibc matcher if a MBCSET is found This patch enables full support of equivalence classes and multicharacter collation symbols. It can also improve performance problems in some cases for multibyte grep. Both of these changes however depend on the glibc version installed in the system. For UTF-8 it will trigger only in the presence of MBCSET, e.g. [a-z]. For other character sets all brackets and `.` as well will trigger it. * NEWS: Document this. * src/dfa.c (dfaexec): Fall back to glibc for multibyte matches, if possible. 2010-09-13 Paolo Bonzini build: update gnulib submodule to latest This is done to include commit "regex: Pass the system regex if its only problem is 32-bit regoff_t". * gnulib: Update to e2b0e1a. 2010-09-12 Jim Meyering build: update gnulib submodule to latest tests: update init.sh from gnulib * tests/init.sh: Update from gnulib. 2010-09-08 Patrick Boyd dfa: reduce stack usage * src/dfa.c (dfaanalyze): Allocate GRPS and LABELS arrays from heap, not on the stack. With this change, grep can now run in these UEFI simulators: http://sourceforge.net/apps/mediawiki/tianocore/index.php?title=EDK http://sourceforge.net/apps/mediawiki/tianocore/index.php?title=EDK2 2010-09-08 Jim Meyering tests/portability: avoid spurious failure with OpenBSD's /bin/sh * tests/warn-char-classes: Don't use "set -x" here. It causes a spurious test failure on openbsd 4.7 when using its /bin/sh, since the command, /bin/sh -xc 'P=1 : 2> err' emits "P=1" into err. To enable set -x, run the test with "VERBOSE=yes", e.g., make check -C tests TESTS=warn-char-classes VERBOSE=yes 2010-09-07 Jim Meyering build: update gnulib submodule to latest 2010-09-03 Jim Meyering tests: remove .sh suffix from remaining test scripts. * tests/backref: Rename from backref.sh. * tests/bre: Rename from bre.sh. * tests/ere: Rename from ere.sh. * tests/file: Rename from file.sh. * tests/khadafy: Rename from khadafy.sh. * tests/options: Rename from options.sh. * tests/pcre: Rename from pcre.sh. * tests/spencer1: Rename from spencer1.sh. * tests/spencer2: Rename from spencer2.sh. * tests/status: Rename from status.sh. * tests/yesno: Rename from yesno.sh. * tests/Makefile.am: Reflect renamings. tests: convert remaining tests to use init.sh * tests/file.sh: Use init.sh. Use Exit, not exit. Use grep, not ${GREP}. * tests/khadafy.sh: Likewise. * tests/options.sh: Likewise. * tests/spencer1.sh: Likewise. * tests/spencer2.sh: Likewise. * tests/status.sh: Likewise. * tests/spencer1.awk: Use grep, not ${GREP}. Don't ignore failure to generate intermediate shell script. * tests/Makefile.am (CLEANFILES): Remove altogether, now that all tests use init.sh. (TESTS_ENVIRONMENT): Don't set GREP. It's no longer used. tests: remove warning.sh * tests/warning.sh: Remove file. All it did was print a warning. * tests/Makefile.am (TESTS): Remove warning.sh. tests: convert pcre.sh to use init.sh * tests/pcre.sh: Use init.sh. Use Exit, not exit. Use grep, not ${GREP}. tests: convert bre.sh to use init.sh * tests/bre.sh: Use init.sh. Use Exit, not exit. Use "$abs_top_srcdir/tests/", not "$srcdir/" to specify inputs. Source generated bre.script, rather than invoking $SHELL. * tests/ere.sh: Likewise. * tests/bre.awk: Use grep, not ${GREP}. * tests/ere.awk: Likewise. * tests/Makefile.am (CLEANFILES): Remove bre.script and ere.script. tests: convert to use init.sh * tests/yesno.sh: Use init.sh. Use Exit, not exit. Use grep, not $GREP. * tests/backref.sh: Likewise. * tests/Makefile.am (CLEANFILES): Remove yesno.txt. build: update gnulib submodule to latest build: update build/test tools from gnulib * bootstrap: Update from gnulib. * tests/init.sh: Likewise. 2010-09-01 Jim Meyering maint: add lib/version-etc.c to the list in POTFILES.in * po/POTFILES.in: Add lib/version-etc.c. 2010-09-01 Jim Meyering grep: diagnose and exit-2 for bogus REs like [:space:], [:digit:], etc. When I make a mistake like this: grep '[:lower:]' ... be it in a script or on the command line, I want to know about it as soon as possible. I don't want grep to print a mere warning that it is interpreting this suspicious and almost guaranteed-wrong regular expression as a set of just 6 bytes. And I certainly don't want grep to silently do the wrong thing, even if that would be officially standards-conforming. It's obvious that I intended [[:lower:]], and I want my error to be diagnosed in a way that is most likely to get my attention. Thus, with this change, grep now prints a diagnostic and exits with status 2 the moment it encounters an offending [:char_class:] construct. This changes the way grep works by default, rather than putting this new behavior on an option. A new option would seldom be used in scripts (not portable), and would probably be used only rarely by those who need it the most. This new functionality provides a valuable safety measure and incurs truly negligible risk. For strict POSIX compliance, set POSIXLY_CORRECT in your environment. That disables this new feature. Revert the changes from commit 2cd3bcea, "grep: add --warnings={always,never,auto}.", and then do the following: * src/dfasearch.c (dfawarn): Call getenv("POSIXLY_CORRECT") here; Remove "warning: " from the diagnostic, now that it's more than a warning, and exit with status 2. * NEWS (New features): Describe the new semantics. * tests/warn-char-classes: Adjust one test to accommodate this change. * doc/grep.texi (Character Classes and Bracket Expressions): Document. (Environment Variables): Cross-reference it. Remove reference to obsolete getopt illegal vs. invalid difference. Thanks to Paul Eggert for suggestions and an initial prod. 2010-08-30 Jim Meyering maint: use gnulib's standard --version-printing code This includes author names and keeps the copyright year up to date. * bootstrap.conf (gnulib_modules): Add propername and version-etc-fsf. * src/main.c (AUTHORS): Define. (main): Use version_etc, rather than hard-coding the copyright text. Prompted by a patch from Paolo Bonzini. 2010-08-27 Paolo Bonzini dfa: warn on [:space:] and similar * src/dfa.c (parse_bracket_exp): Warn on regular expressions such as [:space:]. * src/dfa.h (dfawarn): New prototype. * src/dfasearch.c (dfawarn): New. * NEWS: Document. tests: add test for warnings * tests/Makefile.am (TESTS): Add warn-char-class. * tests/warn-char-class: New. grep: add --warnings={always,never,auto}. * src/grep.h (no_warnings): New declaration. * src/main.c (no_warnings): New. (WARNINGS_OPTION): Add to enum. (main): Add --warnings. Handle color_option == 2 together with it. tests: add failing test for grep from a directory * tests/Makefile.am (TESTS, XFAIL_TESTS): Add grep-dir. * tests/grep-dir: New. tests: add test for previous commit * tests/Makefile.am (TESTS): Add grep-dev-null. * tests/grep-dev-null: New. search: fix "grep -Fif /dev/null" * bootstrap.conf: Include gnulib module minmax. * src/searchutils.c (mbtolower): Handle *N == 0 case. * src/system.h: Include minmax.h from gnulib. 2010-08-27 Adam Katz Remove declaration after statement in dfa.c * dfa.c (dfaexec): Declare saved_end at the beginning of the function. 2010-08-13 Jim Meyering make --include=FILE work once again The semantics of excluded_file_name changed (when operating on an "included" file name list). * src/main.c (main): Adjust for changed semantics of excluded_file_name simply by removing a negation. * NEWS (Bug fixes): Mention this fix. * tests/include-exclude: Add a test for this. Reported by Joe Perches in http://savannah.gnu.org/bugs/?29876. 2010-07-16 Paolo Bonzini doc: document \s and \S * doc/grep.texi (The Backslash Character and Special Expressions): Document \s and \S escapes. 2010-05-29 Karl Berry doc: discuss matches that span two or more lines * doc/grep.texi (Usage): Discuss matching across lines. (Character Classes and Bracket Expressions) <[:space:]>: refer to it. 2010-05-25 Jim Meyering build: use latest gettext: 0.18 * configure.ac: Use gettext-0.18. * bootstrap.conf (gnulib_modules): Use gettext-h, not gettext. since the latter drags in a depedency on gettext 0.18. Suggested by Bruno Haible. maint: update helper scripts from gnulib * tests/init.sh: Update from gnulib. * bootstrap: Likewise. build: update gnulib submodule to latest maint: don't emit an extra newline in each of two diagnostics * src/main.c (context_length_arg, grepdir): Remove a stray \n in each of two diagnostics. 2010-05-24 Bruno Haible search: Avoid out-of-bounds access. * src/dfasearch.c (EGexecute): Avoid access beyond end of buffer that could happen if start != beg - buf. 2010-05-23 Aharon Robbins dfa: fix signedness warnings * src/dfa.c (dfaexec): Cast p when passing it to prepare_wc_buf. 2010-05-09 Jim Meyering tests: update init.sh * tests/init.sh: Update from gnulib. tests: normalize init.sh-sourcing code * tests/backref-multibyte-slow: Use one-line idiom. * tests/backref-word: Likewise. * tests/case-fold-backref: Likewise. * tests/case-fold-backslash-w: Likewise. * tests/case-fold-char-class: Likewise. * tests/case-fold-char-range: Likewise. * tests/case-fold-char-type: Likewise. * tests/char-class-multibyte: Likewise. * tests/dfaexec-multibyte: Likewise. * tests/empty: Likewise. * tests/euc-mb: Likewise. * tests/fedora: Likewise. * tests/fgrep-infloop: Likewise. * tests/fmbtest: Likewise. * tests/foad1: Likewise. * tests/ignore-mmap: Likewise. * tests/include-exclude: Likewise. * tests/max-count-vs-context: Likewise. * tests/pcre-z: Likewise. * tests/prefix-of-multibyte: Likewise. * tests/reversed-range-endpoints: Likewise. * tests/sjis-mb: Likewise. * tests/spencer1-locale: Likewise. * tests/word-delim-multibyte: Likewise. * tests/word-multi-file: Likewise. tests: update help-version * tests/help-version: Update from coreutils. 2010-05-06 Jim Meyering tests: enable glibc's malloc-perturbing option * tests/Makefile.am (MALLOC_PERTURB_): Define, in case it's not already set in your environment. (TESTS_ENVIRONMENT): Propagate MALLOC_PERTURB_ setting to test scripts. 2010-05-06 Paolo Bonzini dfa: speed up [[:digit:]] and [[:xdigit:]] There's no "multibyte pain" in these two classes, since POSIX and ISO C99 mandate their contents. Time for "./grep -x '[[:digit:]]' /usr/share/dict/linux.words" Before: 1.5s, after: 0.07s. (sed manages only 0.5s). * src/dfa.c (predicates): Declare struct dfa_ctype separately from definition. Add sb_only. (find_pred): Return const struct dfa_ctype *. (parse_bracket_exp): Return const struct dfa_ctype *. Do not fill MBCSET for sb_only character types. 2010-05-05 Jim Meyering tests: readability: use awk rather than obfuscated sed * tests/backref-multibyte-slow: Generate input using an awk for-loop rather than expensive and harder-to-read sed pipes. Remove stray "set -x" and "wc -l in". dfa: avoid segfault when processing an invalid multi-byte sequence * src/dfa.c (dfaexec): Handle the cases in which mbrtowc returns (size_t)-1 or (size_t)-2, rather than setting mblen_buf[i] to an outrageously large value. 2010-05-05 Paolo Bonzini grep: remove redundant syntax bit * grep.c (Gcompile): Remove RE_HAT_LISTS_NOT_NEWLINE. tests: add test for newly-fixed performance problem * tests/backref-multibyte-slow: New. * tests/Makefile.am: Add it. 2010-05-05 Paolo Bonzini dfa: convert to wide character line-by-line This provides a nice speedup for -m in general, but especially it avoids quadratic complexity in case we have to go to glibc. * NEWS: Document change. * src/dfa.c (prepare_wc_buf): Extract out of dfaexec. Convert only up to the next newline. (dfaexec): Exit multibyte processing loop if past buf_end. Call prepare_wc_buf again after processing a newline. 2010-05-01 Jim Meyering maint: remove useless #if HAVE_STDLIB_H * src/mbsupport.h: Don't test HAVE_STDLIB_H. 2010-04-20 Jim Meyering dfa: don't #ifdef-out member declarations * src/dfa.c (struct dfa): Remove "#if MBS_SUPPORT" guard that made several member declarations conditional on this cpp definition. (token): Likewise. Reported by Anders Wallin. tests: ensure that the --mmap option is ignored * tests/ignore-mmap: New file. * tests/Makefile.am (TESTS): Add it. Reported by Jaroslav Škarvada in 2010-04-20 Paolo Bonzini dfa: honor RE_DOT_NEWLINE and RE_DOT_NOT_NULL in UTF-8 period optimization * src/dfa.c (add_utf8_anychar): Check for RE_DOT_NEWLINE and RE_DOT_NOT_NULL. grep: fix --mmap not being ignored * NEWS: Document bugfix. * main.c (main): Ignore MMAP_OPTION. 2010-04-19 Jim Meyering maint: avoid syntax-check failure due to indentation via TABs * src/dfa.c (atom): Expand TABs in indentation. build: update gnulib submodule to latest maint: restrict scope of two globals to dfasearch.c * src/dfasearch.c (patterns, pcount): Declare these file-scoped globals to be static. 2010-04-19 Paolo Bonzini dfa: optimize UTF-8 period * NEWS: Document improvement. * src/dfa.c (struct dfa): Add utf8_anychar_classes. (add_utf8_anychar): New. (atom): Simplify if/else nesting. Call add_utf8_anychar for ANYCHAR in UTF-8 locales. (dfaoptimize): Abort on ANYCHAR. dfa: drop ORTOP * src/dfa.c (token, prtok, addtok_mb, nsubtoks, dfaanalyze, dfamust): Remove ORTOP. (regexp): Remove parameter, always add OR at the end, adjust callers. (atom): Adjust caller. (dfaparse): Adjust caller. Always add OR at the end. dfa: fix {0,0} * NEWS: Document change. * src/dfa.c (struct dfa): Remove "broken" field. (lex): Do not set it. (closure): On {0,0}, backup and lex another closure without adding a CAT. (dfabroken): Remove. * src/dfa.h (dfabroken): Remove. * tests/spencer1.tests: Add testcases for {m,n}. dfa: simplify dfainit * src/dfa.c (dfainit): Use memset. 2010-04-17 Jim Meyering doc: fix a nit in HACKING * HACKING: Correct size of .git/ dir: 9MB, not 30MB. tests: add an expected-to-fail test using \< in a multi-byte locale * tests/word-delim-multibyte: New test. Currently failing. * tests/Makefile.am (TESTS): Add it. (XFAIL_TESTS): Define, temporarily. Reported by Jaroslav Škarvada in http://savannah.gnu.org/bugs/?29537. 2010-04-16 Paolo Bonzini test: cover just-fixed bug * tests/empty: Test -Fw too. grep: fix matching the empty string with grep -Fw * NEWS: Document fix. * src/kwsearch.c (Fexecute): The empty string is a valid match if it is a whole word. 2010-04-15 Jim Meyering maint: update init.sh and HACKING * HACKING: Sync from coreutils. * tests/init.sh: Update from gnulib. 2010-04-13 Jim Meyering build: update gnulib submodule to latest; adapt * COPYING: Remove empty line. * README: Likewise. * doc/fdl.texi: Likewise. * tests/backref-word: Likewise. 2010-04-11 Stefano Lattarini tests: accept the Debian timeout program * tests/init.cfg: test timeout with `timeout 10s true' 2010-04-08 Jim Meyering dfa: convert "cannot happen" code/comment to use assert * src/dfa.c (dfamust): There were numerous "cannot happen" comments, some associated with "if (expr) goto done;". Replace each with an equivalent "assert (!expr);". build: use gnulib's isblank module * bootstrap.conf (gnulib_modules): Use gnulib's isblank module, now that we rely on the function by that name. maint: undo TAB-conversion change to gl/lib/*.c.diff This fixes a bootstrap failure due to the patches not applying. * .x-sc_prohibit_tab_based_indentation: Add ^gl/lib/.*\.c\.diff$ * gl/lib/regcomp.c.diff: Revert today's TAB->space change. * gl/lib/regex_internal.c.diff: Likewise. * gl/lib/regexec.c.diff: Likewise. 2010-04-08 Arnold D. Robbins dfa: fix declaration of dfabroken in dfa.h * dfa.h (dfabroken) [GAWK]: Fix declaration to match that in dfa.c. 2010-04-08 Jim Meyering maint: add syntax-check rule to enforce the new no-leading-TABs policy * cfg.mk (sc_prohibit_tab_based_indentation): New rule, from coreutils. (sc_prohibit_emacs__indent_tabs_mode__setting): Likewise. (old_NEWS_hash): Update. * .x-sc_prohibit_tab_based_indentation: List exempt files. 2010-04-08 Jim Meyering convert all TABs to equivalent spaces in indentation Using this file, cat > leading-blank.exempt <<\EOF (?:^|\/)ChangeLog[^/]*$ (?:^|\/)(?:GNU)?[Mm]akefile[^/]*$ \.(?:am|mk)$ EOF run this command to convert all non-conforming leading white space to be all spaces: git ls-files \ | pcregrep -vf leading-blank.exempt \ | xargs pcregrep -l '^ *\t' \ | xargs perl -MText::Tabs -ni -le \ '$m=/^( *\t[ \t]*)(.*)/; print $m ? expand($1) . $2 : $_' 2010-04-08 Jim Meyering build: include cfg.mk in the distribution tarball * Makefile.am (EXTRA_DIST): Add cfg.mk. 2010-04-08 Jim Meyering maint: Makefile.am tweak (no semantic change) * Makefile.am (EXTRA_DIST): List one per line. Sort. build: include cfg.mk in the distribution tarball * Makefile.am (EXTRA_DIST): Add cfg.mk. 2010-04-08 Jim Meyering dfa: move definition of __attribute__ back into dfa.h * src/dfa.c (__attribute__): Move definition back to... * src/dfa.h: ... this file. It is essential for non-gcc compilers. Reported by Arnold Robbins. 2010-04-07 Arnold D. Robbins dfa: move internals from dfa.h to dfa.c * src/dfa.h: Move internals into dfa.c. * src/dfa.c: The dfa internals are now totally local to this file. (dfaalloc, dfamusts, dfabroken): New functions to access features. * src/dfasearch.c (dfa): Change this global variable from struct to pointer. Adapt to that change, and use new functions, dfamusts and dfaalloc. 2010-04-07 Jim Meyering mbtolower: avoid potential NULL-dereference * src/searchutils.c: Include . (mbtolower): Assert that 0 < *n, to avoid possibility of NULL-deref. Remove dead increment. maint: tell git to ignore more build products * .gitignore: Also ignore results of "make ID" and "make tags". build: update gnulib submodule to latest tests: use init.sh consistently * tests/euc-mb: Call "path_prepend_ ." on a line by itself, and with a comment. This makes it so all of the srcdir/init.sh lines are consistent, project-wide, and so that the addition of "." to PATH for this test is properly documented. * tests/sjis-mb: Likewise. maint: avoid new syntax-check failure, ... ...now that the sole use of xmalloc no longer matches the regular expression used by the syntax-check rule. * .x-sc_prohibit_xalloc_without_use: Exempt src/kwset.c. grep: make kwset's obstack use xmalloc, not malloc This insidious bug could make grep fail to diagnose a failed malloc, and then proceed to dereference the resulting NULL pointer. Note that this bug was unlikely ever to cause real trouble; without the fix, grep would segfault upon OOM, now it exits with a diagnostic. * src/kwset.c (malloc) [GREP]: Define without the "(s)" macro parameter, so that unadorned uses of malloc are also mapped to xmalloc. One such use is in the expansion of obstack_init. Report and patch by Nelson H. F. Beebe, in http://thread.gmane.org/gmane.comp.gnu.grep.bugs/2995 tests: improve help-version (sync from gzip's version) * tests/help-version: Cross-check $VERSION and --version output. * tests/Makefile.am (TESTS_ENVIRONMENT): Export VERSION=$(VERSION). 2010-04-06 Jim Meyering doc: update THANKS * THANKS: Update. 2010-04-06 Aharon Robbins build: avoid conflict with WCHAR definition from Cygwin's * src/dfa.h (enum token): Remove the definition from this file. Replace with a declaration and typedef. Moved to ... * src/dfa.c (enum token): ... here. Reported by Corinna Vinschen. 2010-04-06 Jim Meyering doc: add HACKING * HACKING: New file. Copied from coreutils, with s/coreutils/grep/ and a few minor edits. 2010-04-05 Jim Meyering tests: pull fixed init.sh from gnulib * tests/init.sh: Update from gnulib. maint: fix new argmatch-related syntax-check failures * configure.ac (ARGMATCH_DIE): Use usage(EXIT_FAILURE), not exit(1). * po/POTFILES.in: Add lib/argmatch.c. maint: update cfg.mk to work with gnulib's newer "make syntax-check" * cfg.mk: Update to use new _sc_search_regexp interface. Run this: perl -pi -e 's/\b_prohibit_regexp\b/_sc_search_regexp/;' -e 's/\bmsg=/halt=/; s/\bre=/prohibit=/;' cfg.mk and then adjust backslashes so they still line up. maint: update tests/init.sh from gnulib This ensures that the explanation for any skipped or failed test is printed on stderr, not buried in each .log file. * tests/init.sh: Update from gnulib. * tests/init.cfg (stderr_fileno_): Define to 9, to match the literal 2>&9 in tests/Makefile.am build: update gnulib submodule to latest 2010-04-04 Jim Meyering maint: use argmatch, for better --directories=INVAL diagnostics Before, you'd see this: grep: unknown directories method Now, you'll see this: grep: invalid argument `INVAL' for `--directories' Valid arguments are: - `read' - `recurse' - `skip' Usage: src/grep [OPTION]... PATTERN [FILE]... Try `src/grep --help' for more information. * bootstrap.conf: Add argmatch. * configure.ac: Define ARGMATCH_DIE and ARGMATCH_DIE_DECL. * src/main.c (directories_type): Define. (directories_args, directories_types) Define. All of the above so we can... (main): Use XARGMATCH. (usage): Declare extern, now that argmatch calls it via ARGMATCH_DIE. 2010-04-04 Jim Meyering dfa.c: const correctness; and remove useless casts of realloc and malloc * src/dfa.c (icatalloc, icpyalloc, istrstr, enlist): As above. (inboth, dfamust, comsubs): Likewise. dfa.c: use a better (unsigned) type for an index: int->unsigned int * src/dfa.c (dfaexec): Use "unsigned int" for a logically unsigned index. maint: style: use sizeof VAR, rather than sizeof TYPE, where possible * src/dfa.c (copyset, zeroset): Prefer sizeof EXPR, over sizeof TYPE, for improved readability/maintainability. (equal, parse_bracket_exp, addtok_wc, dfaparse, dfaexec): Likewise. 2010-04-02 Jim Meyering dfa.c: use a better (unsigned) type for an index: int->size_t * src/dfa.c (parse_bracket_exp): Use size_t as type of index, not int. maint: const-correctness * src/dfa.c (tstbit, copyset, equal, charclass_index): Declare read-only "charclass" parameters to be "const". No semantic change. maint: include and unconditionally * src/main.c: Include and unconditionally. Their presence/usefulness are assured by gnulib. * src/dfa.c: Likewise. * src/search.h: Likewise. maint: MBS_SUPPORT: define to 0/1, not undef/1 Prepare to remove many of these #ifdefs. * src/mbsupport.h (MBS_SUPPORT): Define to 0/1, not undef/1. Change each "#ifdef MBS_SUPPORT" to "#if MBS_SUPPORT". Use this: perl -pi -e 's/ifdef (MBS_SUPPORT)/if $1/' $(g grep -l ifdef.MBS_SUPPO) * src/dfa.c: s/#ifdef MBS_SUPPORT/#if MBS_SUPPORT/ * src/dfa.h: Likewise. * src/dfasearch.c: Likewise. * src/kwsearch.c: Likewise. * src/main.c: Likewise. * src/search.h: Likewise. * src/searchutils.c: Likewise. 2010-04-02 Jim Meyering maint: use STREQ in place of strcmp perl -pi -e 's/\bstrcmp *\((.*?)\) == 0/STREQ ($1)/' src/main.c perl -pi -e 's/\bstrcmp *\((.*?)\) != 0/!STREQ ($1)/' src/main.c * src/dfa.c (STREQ): Define. Use it instead of strcmp. * src/main.c (STREQ): Likewise. * cfg.mk (local-checks-to-skip): Remove sc_prohibit_strcmp, to enable the strcmp-prohibition. 2010-04-02 Jim Meyering maint: enable the useless_cpp_parens syntax check * cfg.mk (local-checks-to-skip): Remove sc_useless_cpp_parens. * src/main.c (devices, fillbuf, exit_on_match): Remove useless parens. (print_line_head, grepfile, set_limits, main): Likewise. * src/vms_fab.h: Likewise. * vms/config_vms.h: Likewise. * src/mbsupport.h: Likewise. cleanup and improvement: parse command line arguments consistently * src/main.c: Include c-ctype.h, for this: (prepend_args): Use c_isspace, not ISSPACE. This is important so that we parse arguments consistently, and independently of the current locale. * bootstrap.conf (gnulib_modules): Add c-ctype. * src/system.h: Remove IS* definitions here, too. * src/dfasearch.c (WCHAR): Use isalnum, not ISALNUM. * src/kwsearch.c (WCHAR): Likewise. * src/searchutils.c (kwsinit): Use tolower, not TOLOWER. cleanup: rely on gnulib's ctype.h functions; remove IS* macros and is_* * src/dfa.c (setbit_case_fold, prednames): Use official names. (IS_WORD_CONSTITUENT, lex): Likewise. (ISALNUM, ISALPHA, ISCNTRL, ISDIGIT, ISGRAPH): Remove definitions. (ISLOWER, ISPRINT, ISPUNCT, ISSPACE, ISUPPER, ISXDIGIT): Likewise. (is_alnum, is_alpha, is_blank, is_cntrl, is_digit, is_graph): Likewise. (is_lower, is_print, is_punct, is_space, is_upper, is_xdigit): Likewise. (isgraph): Likewise. build: update gnulib submodule to latest, and adjust * src/main.c (parse_grep_colors): Adjust diagnostics not to trigger the sc_error_message_period and sc_error_message_uppercase syntax-check rules. maint: remove all VMS-related code * configure.ac (AC_CONFIG_FILES): Remove vms/Makefile * Makefile.am (SUBDIRS): Remove vms. * src/Makefile.am (EXTRA_DIST): Remove vms_fab.c and vms_fab.h. * src/vms_fab.c, src/vms_fab.h, vms/make.com: Remove files. * vms/Makefile.am, vms/README, vms/config_vms.h: Likewise. post-release administrivia * NEWS: Add header line for next release. * .prev-version: Record previous version. * cfg.mk (old_NEWS_hash): Auto-update. version 2.6.3 * NEWS: Record release date. 2010-04-02 Jim Meyering grep: avoid used-undefined error with truncated multibyte input * src/dfa.c (addtok_wc): Don't use buf[0] (it's undefined) when wcrtomb returns <= 0. MBS_SUPPORT-removal: * src/dfa.c (dfastate): 2010-04-01 Jim Meyering maint: avoid unnecessary 2nd getenv("TERM") * src/main.c (main): Don't call getenv("TERM") twice -- in the same expression, even. tests: remove all unportable uses of echo * src/main.c: Use printf rather than echo -ne in a comment. * tests/fedora: Use printf (not echo) also in ok/fail functions. * cfg.mk (sc_prohibit_echo_minus_en): New rule, to prohibit any future introduction. tests: add explicit requirement for en_US.UTF-8 * tests/char-class-multibyte: Use require_en_utf8_locale_, rather than open-coding it. * tests/prefix-of-multibyte: Require the locale explicitly. * tests/fgrep-infloop: Likewise. This fixes test failures that would arise on systems without that particular locale. Reported by Ludovic Courtès. tests: new function, to require an en_US UTF8 locale * tests/init.cfg (require_en_utf8_locale_): New function. tests: use printf, not echo -n, echo -e, or any combination * tests/fedora: Using printf is more portable. grep: remove unnecessary code * src/main.c (print_line_middle): Now that we use RE_ICASE (enabled in commit 70e23616, "dfa: rewrite handling of multibyte case_fold lexing"), this case-conversion code is useless and wasteful. Remove it. doc: fix typo: s/AM_V_AT/AM_V_at/ * doc/Makefile.am (egrep.1 fgrep.1): The former has case consistent with its sister variable, AM_V_GEN, but the latter is the one that actually works. doc: generated files are best made read-only, ... ...to minimize risk of accidentally modifying the generated file rather than its template. These are tiny, so no risk, but it's a good to be consistent, so generated files are easier to spot. * doc/Makefile.am (egrep.1 fgrep.1): When generating these files, ensure that they too are created read-only. doc: generate grep.1 from template * doc/Makefile.am (grep.1): New rule. (CLEANFILES): Add grep.1 to the list. * .gitignore: Add /doc/grep.1 * doc/grep.in.1: Replace hard-coded "2.5.1-cvs" with @VERSION@. Update copyright year list. Omit the line-splitting \(co directive so that update-copyright will perform future updates automatically. Egmont Koblinger reported the outdated version string and copyright year list in the man page: http://savannah.gnu.org/bugs/?29390 doc: prepare to generate grep.1 * doc/grep.1: Rename to... * doc/grep.in.1: ...this. 2010-03-31 Eric Blake build: avoid another warning Noticed on cygwin: get-mb-cur-max.c: In function 'main': get-mb-cur-max.c:27: error: unused parameter 'argc' [-Wunused-parameter] * tests/get-mb-cur-max.c (main): Use argc. 2010-03-31 Paolo Bonzini tests: fix on systems with broken sh * tests/Makefile.am (TESTS_ENVIRONMENT): Adjust coreutils remnants. * tests/bre.sh: Invoke script with $SHELL if defined. * tests/ere.sh: Likewise. * tests/spencer1-locale: Likewise. * tests/spencer1.sh: Likewise. tests: improve empty test * tests/empty: Add more tests, note expected failure. tests: improve empty test with respect to locales * tests/empty: Add tests for multiple locales. grep: fix grep -F against empty string * src/searchutils.c (is_mb_middle): Do not return true for empty matches when p == buf. tests: rename empty.sh to empty * tests/empty.sh: Rename to... * tests/empty: ... this. * tests/Makefile.am (TESTS): Adjust. tests: convert empty.sh to new style * tests/empty.sh: Convert to init.sh, add 10-second timeout. tests: use get-mb-cur-max in char-class-multibyte * tests/char-class-multibyte: Use get-mb-cur-max to detect UTF-8 support. Rewrite previous locale detection code as a grep test. tests: fix -Wformat failure * tests/get-mb-cur-max (main): Cast MB_CUR_MAX to int. 2010-03-30 Jim Meyering doc: add a "Reply-To" to the suggested announcement mail header * README-release: Add "Reply-To" with the list address, to minimize risk of replies to the other announcement recipients. Suggestion from Eric Blake. 2010-03-29 Jim Meyering build: avoid compiler warning when building test program * tests/Makefile.am (AM_CPPFLAGS, AM_CFLAGS, AM_LDFLAGS): Define, so that all the usual C compile-and-link machinery comes into play. * tests/get-mb-cur-max.c: Include "progname.h". Remove unnecessary inclusion of . Mike Frysinger reported the "implicit decl of set_program_name" warning. build: detect PCRE support also when is the header * m4/pcre.m4: Also check for . * src/pcresearch.c: Include , if needed. Guard inclusions with HAVE_PCRE_H and HAVE_PCRE_PCRE_H, not HAVE_LIBPCRE. * NEWS (Bug fixes): Mention it. Dmitry V. Levin reported that PCRE support was not detected on systems with not in the default include path. post-release administrivia * NEWS: Add header line for next release. * .prev-version: Record previous version. * cfg.mk (old_NEWS_hash): Auto-update. version 2.6.2 * NEWS: Record release date. 2010-03-29 Eric Blake build: avoid warnings on cygwin * lib/savedir.c (isdir): Avoid shadowing a declaration. * src/main.c (get_nondigit_option): Cast away const to avoid compiler warning. maint: ignore new test executable * .gitignore: Enhance. 2010-03-29 Jim Meyering doc: consolidate redundant-looking entries * NEWS: Consolidate the two --include/exclude-related entries. Suggested by Eric Blake. 2010-03-29 Paolo Bonzini tests: use $(...) consistently * tests/backref.sh: Use `...' instead of ``...'' in comments. * tests/bre.awk: Use $(...) instead of `...`. * tests/ere.awk: Use $(...) instead of `...`. * tests/euc-mb: Use $(...) instead of `...`. * tests/fmbtest: Use $(...) instead of `...`. * tests/foad1: Use $(...) instead of `...`. * tests/pcre-z: Use $(...) instead of `...`. Quote output of grep. * tests/spencer1-locale.awk: Use $(...) instead of `...`. * tests/spencer1.awk: Use $(...) instead of `...`. * tests/yesno.sh: Use $(...) instead of `...`. 2010-03-29 Jim Meyering build: make doc/Makefile.am cleaner and more robust * doc/Makefile.am (egrep.1 fgrep.1): Generate robustly, i.e., do not redirect directly to $@. Use $(AM_V_GEN). Do not distribute intermediate files like fgrep.man and egrep.man. Likewise, do not use them to generate their %.1 images. Instead, generate the .1 files directly. 2010-03-29 Paolo Bonzini tests: add program to detect locales * tests/Makefile.am (check_PROGRAMS): Add get-mb-cur-max. * tests/get-mb-cur-max.c: New. * tests/euc-mb: Use it. Fail if the former detection test fails. * tests/sjis-mb: Use it. Fail if the former detection test fails. Expand comments. 2010-03-29 Paolo Bonzini tests: add tests for SJIS character sets The attached test will be skipped unless (on a glibc system) you run something like mkdir /usr/lib/locale/ja_JP.SHIFT_JIS zcat /usr/share/i18n/charmaps/SHIFT_JIS.gz | \ localedef \ -f - \ -i /usr/share/i18n/locales/ja_JP \ /usr/lib/locale/ja_JP.SHIFT_JIS * tests/Makefile.am: Add sjis-mb. * tests/sjis-mb: New. 2010-03-29 Paolo Bonzini grep -F: fix a bug with SJIS character sets Commit db9d6 would erroneously skip matches in SJIS character sets. In this character set low bytes (i.e. ASCII bytes) are also valid second bytes in a double-byte character, so you have to continue looking for a match, even if you match in the middle of a double-byte character. * src/kwsearch.c: Ensure that beg is advanced by at least one byte, but do not fail immediately after matching in the middle of a double-byte character. 2010-03-28 Bruno Haible build: update after change in gnulib's lib-ignore module * src/Makefile.am (AM_LDFLAGS): Define. Use gnulib's new $(IGNORE_UNUSED_LIBRARIES_CFLAGS). 2010-03-28 Jim Meyering tests: disable new texinfo-acronym syntax-check from gnulib * cfg.mk (local-checks-to-skip): Add new sc_texinfo_acronym, to skip it. 2010-03-28 Norihiro Tanaka tests: exercise fix for improper match of incomplete MB char prefix * tests/prefix-of-multibyte: New file. * tests/Makefile.am (TESTS): Add it. 2010-03-28 Jim Meyering grep -F: fix a multi-byte erroneous-match-in-middle bug Just as Perl prints nothing in this case, printf '\357\274\241\n' | perl -CIO -lne '/\357/ and print' grep should also print nothing when used as follows. However, these would mistakenly match with grep prior to 2.6.2: printf '\357\274\241\n' | LC_ALL=en_US.UTF-8 src/grep -F $'\357' printf '\357\274\241\n' | LC_ALL=en_US.UTF-8 src/grep -F $'\357\274' * src/searchutils.c (is_mb_middle): New parameter: the length of the match, in bytes, as determined by kwsexec. Use this to detect when the nominal match found by kwsexec must be skipped because it is for an incomplete multi-byte character that is a prefix of a character in the input. * src/dfasearch.c (EGexecute): Update caller. * src/kwsearch.c (Fexecute): Likewise. * src/search.h: Update prototype. * NEWS (Bug fixes): Mention it. Report and analysis by Norihiro Tanaka. 2010-03-28 Norihiro Tanaka tests: add tests for the fgrep-infloop bug * tests/init.cfg (require_timeout_): New function. * tests/fgrep-infloop: New file. Test for the above fix. * tests/Makefile.am (TESTS): Add it. 2010-03-28 Jim Meyering grep -F: avoid infinite loop when searching for incomplete MB character Searching for an incomplete non-prefix of a multi-byte character should find no match. Just as these print nothing, printf '\357\274\241\357\274\241\n' \ | perl -CIO -ne '/\241\357/ and print' printf '\357\274\241\n' | perl -CIO -ne '/\274\241/ and print' printf '\357\274\241\n' | perl -CIO -ne '/\241/ and print' printf '\357\274\241\n' | perl -CIO -ne '/\274/ and print' These should also print nothing, but with grep-2.6 and grep-2.6.1, they would infloop: printf '\357\274\241\n' | LC_ALL=en_US.UTF-8 src/grep -F $'\241' printf '\357\274\241\n' | LC_ALL=en_US.UTF-8 src/grep -F $'\274' printf '\357\274\241\n' | LC_ALL=en_US.UTF-8 src/grep -F $'\274\241' * src/kwsearch.c (Fexecute): Don't infloop when searching for an incomplete non-prefix part of a multi-byte character. * NEWS (Bug fixes): Mention it. Reported and diagnosed by Norihiro Tanaka. 2010-03-28 Jim Meyering tests: rename: fmbtest.sh -> fmbtest * tests/fmbtest.sh: Rename to ... * tests/fmbtest: ...this, dropping the .sh suffix. * tests/Makefile.am (TESTS): Reflect renaming. tests: convert fmbtest.sh to use init.sh * tests/fmbtest.sh: Use init.sh and adapt accordingly: Use "grep", not ${GREP}. Use Exit, not exit. tests: also exercise the --include + glob path * tests/include-exclude: Exercise Javier's fix. 2010-03-28 Javier Villavicencio grep -r: fix --include with globs, too The previous fix addressed only the non-glob case. * src/main.c (main): Use add_exclude's EXCLUDE_WILDCARDS option, to enable the use of fnmatch with --include=GLOB. gnulib: Update to latest, for the fixed exclude.c. 2010-03-28 Jim Meyering grep -r: fix --include with non-globs * lib/savedir.c (savedir): Fix logic error. Introduced by commit bf3bd92c, "build: adapt to the newer exclude API we now get from gnulib" * tests/include-exclude: Test for this bug by exercising --include, too. * NEWS (Bug fixes): Mention it. Reported by Philipp Kohlbecher in http://savannah.gnu.org/bugs/?29358 2010-03-27 Jim Meyering kwset: correct comments; require non-NULL kwsmatch argument * src/kwset.c (kwsexec): Correct comments. This function has been returning an offset, not a pointer, for 9 years. Do not test for kwsmatch == NULL. All callers pass non-NULL. (cwexec): Likewise. * src/kwset.h (kwsexec): Mark the 4th parameter, kwsmatch, as non-NULL. Include "arg-nonnull.h". build: add -I$(top_builddir)/lib so we also find generated .h files * src/Makefile.am (AM_CPPFLAGS): Rename from INCLUDES to avoid warning from automake -Wall. Add -I$(top_builddir)/lib, so we find generated .h files like getopt.h in a non-srcdir build. build: remove superfluous LOCALEDIR definition * src/Makefile.am (INCLUDES): Remove unnecessary definition of LOCALEDIR here. Now, it's defined via gnulib's configmake.h. * src/system.h: Include "configmake.h" for its LOCALEDIR definition. grep: don't segfault upon use of --include or --exclude* options * lib/savedir.c (isdir1): Fix fatal typo: deref "dir" argument, not the global (initially-NULL) "path". Reported by Standish Parsley. * tests/include-exclude: New file. * tests/Makefile.am (TESTS): Add it. * NEWS (Bug fixes): Mention it. 2010-03-26 Jim Meyering tests: rename: foad1.sh -> foad1 * tests/foad1.sh: Rename to ... * tests/foad1: ...this, dropping the .sh suffix. * tests/Makefile.am (TESTS): Reflect renaming. tests: convert foad1.sh to use init.sh This fixes a spurious test failure when "make check" is run with certain envvars set, e.g., "make check GREP_COLOR=always" * tests/foad1.sh: Use init.sh and adapt accordingly: Use "grep", not ${GREP}. Test VERBOSE against "yes", not "1", to be consistent with init.sh. Use Exit, not exit. Reported by Nelson H. F. Beebe. tests: insulate tests from envvar settings * tests/init.cfg (vars_): Unset each envvar that can affect how grep works. This protects only those tests that have been converted to use init.sh. 2010-03-25 Eric Blake maint: ignore 'make dist pdf' droppings * .gitignore: Add more exemptions. 2010-03-25 Jim Meyering tests: avoid spurious test failure due to lack of a French UTF8 locale * tests/init.cfg: New file. If either $LOCALE_FR or $LOCALE_FR_UTF8 is set to "none", reset it to the empty string. Reported by Mike Frysinger and Sven Joachim. * tests/Makefile.am (EXTRA_DIST): Add init.cfg. build: do not use pkg-config to test for PCRE support * configure.ac: Do not use PKG_PROG_PKG_CONFIG or PKG_CHECK_MODULES. Do not modify CPPFLAGS; that belongs to those who invoke make. Instead, use autoconf's AC_CHECK_HEADERS and AC_SEARCH_LIBS via the new macro, gl_FUNC_PCRE, defined in... * m4/pcre.m4 (gl_FUNC_PCRE): New macro, to handle pcre-related configure-time tests. * src/Makefile.am (grep_LDADD): Use LIB_PCRE, not PCRE_LIBS. * src/pcresearch.c: Test HAVE_LIBPCRE via "#if", not "#ifdef". All other cpp tests of this symbol used "#if". Prompted by a suggestion from Bruno Haible. * NEWS (Build-related): Mention this. doc: correct and amend NEWS entries for 2.6.1 * NEWS (Bug fixes): Correct character ranges bug description. Add an example from Dmitry V. Levin. Add that the word-with-backref bug was introduced in 2.5.1. * cfg.mk (old_NEWS_hash): Update to match. post-release administrivia * NEWS: Add header line for next release. * .prev-version: Record previous version. * cfg.mk (old_NEWS_hash): Auto-update. version 2.6.1 * NEWS: Record release date. 2010-03-25 Tony Abou-Assaleh tests: use awk's -v option more portably * tests/spencer1-locale: Add a space between awk's "-v" option and the following VAR=value string, to avoid test failure on Mac OS X. 2010-03-25 Norihirio Tanaka dfa/grep: fix compilation with MBS_SUPPORT * src/dfa.c (cur_mb_len): Initialize to 1 and always make it available. (setbit_case_fold): Do not use wint_t in prototype if !MBS_SUPPORT. (parse_bracket_exp): Fix compilation with !MBS_SUPPORT. * src/kwsearch.c (kwsinit): Do not use mbtolower and MB_CUR_MAX if !MBS_SUPPORT. * src/searchutils.c (kwsinit): Do not refer to MB_CUR_MAX if !MBS_SUPPORT. * tests/char-class-multibyte: Skip if UTF-8 matching does not work. * tests/fmbtest.sh: Likewise. 2010-03-25 Jim Meyering build: avoid warnings about unnecessary use of "return" * src/grep.c (Gcompile, Ecompile, Acompile): Do not "return X" from a function returning void, not even when X itself is a function returning void. This avoids warnings from Sun Studio 11 reported by Dagobert Michelsen. * src/egrep.c (Ecompile): Likewise. 2010-03-25 Norihirio Tanaka grep: fix printing when -w is used and regex is needed for matching * NEWS: Document bugfix. * src/dfasearch.c (EGexecute): After assess_pattern_match len, is either invalid or end-beg; jump to success. * tests/Makefile.am (TESTS): Add new test. * tests/backref-word: New. 2010-03-25 Paolo Bonzini dfa: fix single byte character ranges * src/dfa.c (in_coll_range): Fix ordering for second strcoll. Reported by Dmitry V. Levin. * tests/spencer1-locale.awk: Also test single-byte character sets. * NEWS: Add a note about this bugfix. * THANKS: Add Dmitry. 2010-03-25 Norihirio Tanaka grep: reset state after truncated or invalid multibyte sequences * src/searchutils.c (is_mb_middle): When treating an invalid sequence or a truncated multibyte character as a single byte character, reset mbstate grep: do lowercase conversion in print_line_middle only for single-byte case * src/main.c (print_line_middle): Restrict match_icase code to MB_CUR_MAX == 1. Adjust comments. 2010-03-25 Jim Meyering tests: provide framework_failure_ function The shell function "framework_failure" was called in the unusual event that some fundamental test set-up operation would fail. However it was not defined. Define it, but with a trailing underscore to impinge less on the test writer's name space. Adjust all uses. * tests/init.sh (framework_failure_): New function. * tests/case-fold-backref: s/framework_failure/framework_failure_/ * tests/case-fold-char-class: Likewise. * tests/case-fold-char-range: Likewise. * tests/case-fold-char-type: Likewise. * tests/char-class-multibyte: Likewise. * tests/dfaexec-multibyte: Likewise. * tests/max-count-vs-context: Likewise. * tests/word-multi-file: Likewise. 2010-03-24 Jim Meyering doc: tweak THANKS * THANKS: Update Arnold's name and address, per request. portability: use gnulib's lseek wrapper * bootstrap.conf (gnulib_modules): Use gnulib's lseek wrapper, for improved portability. lseek does not fail with ESPIPE on pipes on some systems. build: avoid link failure on Solaris 8 * bootstrap.conf (gnulib_modules): Add wctob. * NEWS (Portability): Mention this. Reported by Dagobert Michelsen in . 2010-03-24 Petr Písař doc: translate new --help message * src/main.c: Translate "after_options". 2010-03-24 Jim Meyering doc: NEWS make it clear that the bug was introduced in 2.6 * NEWS: Clarify. 2010-03-24 Paolo Bonzini tests: fix char-class-multibyte * tests/char-class-multibyte: Make it pass. 2010-03-23 Jim Meyering build: avoid compilation failure when MBS_SUPPORT not defined * src/dfa.c (setbit_case_fold) [!MBS_SUPPORT]: Fix curly brace mismatch. 2010-03-23 Paolo Bonzini dfa: fix sigsegv on multibyte character classes Reported by Jaroslav Škarvada . This is unfortunate. grep needs an automatic testcase generator. * NEWS: Document bug. * THANKS: Mention reporter. * src/dfa.c (set_bit_casefold): Change type of first argument for self-documentation. (parse_bracket_exp): Fix call. * tests/Makefile.am: Add new testcase. * tests/char-class-multibyte: New testcase. 2010-03-23 Jim Meyering post-release administrivia * NEWS: Add header line for next release. * .prev-version: Record previous version. * cfg.mk (old_NEWS_hash): Auto-update. version 2.6 * NEWS: Record release date. build: avoid warnings: tell gcc and clang that dfaerror never returns * src/dfa.h (__attribute__): Define. (dfaerror): Declare with the "noreturn" attribute. * src/dfasearch.c (dfaerror): Add an unreachable use of abort. 2010-03-22 Eric Blake build: fix cygwin build Portions of gnulib depend on -lintl, and cygwin does not allow lazy linking. * src/Makefile.am (LDADD): Include libraries in correct order. 2010-03-22 Paolo Bonzini grep: remove --mmap mmap is a bad idea for sequentially accessed file because it will cause a page fault for every read page. Just consider it a failed experiment, and ignore --mmap while accepting it for backwards compatibility. * configure.ac (AC_FUNC_MMAP): Remove. * doc/grep.texi (Other options): Say --mmap is ignored. * src/grep.c (mmap_option): Remove. (long_options): Do not reference it. (bufmapped, initial_bufoffset): Remove. (reset, fillbuf): Remove HAVE_MMAP code. (grepfile): Remove bufmapped reference. (usage): Say --mmap is ignored. 2010-03-22 Paolo Bonzini grep: rename files for intuitiveness * Makefile.am (libgrep_a_SOURCES, grep_SOURCES, egrep_SOURCES, fgrep_SOURCES): Adjust. * grep.c: Rename to main.c. * esearch.c: Rename to egrep.c. * fsearch.c: Rename to fgrep.c. * gsearch.c: Rename to grep.c. grep: kill GREP_PROGRAM/EGREP_PROGRAM/FGREP_PROGRAM * NEWS: Document slight semantic change. * TODO: #ifdefs are gone. * po/POTFILES.in: Update. * src/Makefile.am (grep_SOURCES, egrep_SOURCES, fgrep_SOURCES): Remove grep.c/egrep.c/fgrep.c. (noinst_LIBRARIES): Change libsearch.a to libgrep.a. (libsearch_a_SOURCES): Rename to libgrep_a_SOURCES, add grep.c (LDADD): Change libsearch.a to libgrep.a. * src/esearch.c: Add before_options and after_options. * src/fsearch.c: Likewise. * src/gsearch.c: Likewise. * src/grep.c (short_options, long_options): Remove GREP_PROGRAM special-casing. (usage): Use before_options and after_options, look at matchers. (setmatcher): Merge with install_matcher. (main): Call setmatcher (NULL) instead of install_matcher. * src/grep.h (GREP_PROGRAM): Remove. (before_options, after_options): Add. thank Eric Blake * THANKS: Add Eric Blake, who reported the warning fixed by 774d0ee. grep: libify *search.c * src/Makefile.am (libsearch_a_SOURCES): Add dfasearch.c, kwsearch.c, pcresearch.c. * src/esearch.c, src/fsearch.c, * src/gsearch.c: Only include search.h. * src/dfasearch.c (GEAcompile, EGexecute): Export. * src/kwsearch.c (Fcompile, Fexecute): Export. * src/pcresearch.c (Pcompile, Pexecute): Export. * src/search.h: Add new exported functions. grep: prepare for libification of *search.c * src/dfasearch.c (Ecompile): Remove. * src/esearch.c: Place it here... * src/gsearch.c: ... and here. grep: split search.c * po/POTFILES.in: Update. * src/Makefile.am (grep_SOURCES, egrep_SOURCES, fgrep_SOURCES): Move kwset.c and dfa.c to libsearch.a. Add searchutils.c there too. * src/search.h, src/dfasearch.c, src/pcresearch.c, src/kwsearch.c, src/searchutils.c: New files, split out of src/search.c. * src/esearch.c, src/fsearch.c: Include the new files instead of search.c. * src/gsearch.c: Likewise, plus move Gcompile/Acompile here. grep: remove one #ifdef * search.c (GEAcompile) [EGREP_PROGRAM]: Use common code. Inline IF_BK. 2010-03-22 Paolo Bonzini grep: eliminate {COMPILE,EXECUTE}_{RET,ARGS,FCT} Modern compilers warn about type mismatches. * src/grep.c (do_execute): Write full declaration. * src/grep.h (COMPILE_RET, COMPILE_ARGS, COMPILE_FCT, EXECUTE_RET, EXECUTE_ARGS, EXECUTE_FCT): Remove. (compile_fp_t, execute_fp_t): Write full declaration. * src/search.c (GEAcompile, Gcompile, Acompile, Ecompile, EGexecute, Fcompile, Fexecute, Pcompile, Pexecute): Write full declaration. 2010-03-22 Paolo Bonzini grep: make egrep/fgrep use struct matcher * Makefile.am (grep_SOURCES): Add gsearch.c. (EXTRA_DIST): Add search.c. * esearch.c (matchers): New. * fsearch.c (matchers): New. * gsearch.c: New. * search.c (matchers): Remove. * grep.c: Always compile most !GREP_PROGRAM sections. (main): Use first matcher if none is explicitly provided. Remove "default" matcher. * grep.h (struct matcher): Adjust comments. grep: change struct matcher termination * src/grep.c (setmatcher): Look for NULL matchers[i].name. * src/grep.h (struct matcher): Change name to pointer. Adjust comments. * src/search.c (matchers): Terminate with three NULLs. grep: remove one #ifdef * search.c (Ecompile): Always go through GEAcompile to use same code path for both grep and egrep. grep: remove getpagesize.h * src/getpagesize.h: Remove. * src/Makefile.am (noinst_HEADERS): Remove getpagesize.h. 2010-03-21 Jim Meyering build: use the fcntl-h module, not "fcntl" * bootstrap.conf (gnulib_modules): We might need fcntl.h somewhere, but don't use the fcntl function. Reported by Bruno Haible. build: avoid link failure on systems using gnulib's fcntl but not open * bootstrap.conf (gnulib_modules): Using gnulib's fcntl module and including , but not also using gnulib's "open" module would result in link failure due to references to rpl_open on systems requiring the replacement (e.g., Cygwin and Darwin). build: avoid compilation failure on systems using rpl_open This new build failure has arisen as a result of using gnulib's "fcntl" module. Now that an inadequate "open" syscall is replace by gnulib's wrapper, it is essential to include . * src/grep.c: Include . This is required, for grepfile's use of open, at least on Cygwin and Darwin. maint: use gnulib's fcntl module, just in case * bootstrap.conf (gnulib_modules): Add fcntl. Grep uses at least O_BINARY, which may be defined therein. maint: remove TYPE_* definitions from src/system.h * src/system.h (TYPE_MAXIMUM, TYPE_MINIMUM, TYPE_SIGNED): Remove definitions. They are provided by intprops.h. * src/grep.c: Include "intprops.h" * bootstrap.conf (gnulib_modules): Add intprops. maint: alphabetize #include directives * src/grep.c: Alphabetize #include directives. 2010-03-20 Jim Meyering build: stop using gnulib's memmove module * bootstrap.conf (gnulib_modules): Remove obsolete module: memmove build: reinstate gnulib's fcntl-h-tests * bootstrap.conf (gnulib_tool_option_extras): Do not avoid the fcntl-h-tests. I cannot reproduce the failure. 2010-03-20 Eric Blake build: allow compilation on cygwin Gnulib is incompatible with -Wunused-macros. Addtionally, cygwin 1.7.1 coupled with --enable-gcc-warnings tripped on: grep.c: In function 'print_line_middle': grep.c:805: error: array subscript has type 'char' [-Wchar-subscripts] grep.c: In function 'main': grep.c:1833: error: 'optarg' redeclared without dllimport attribute: previous dllimport ignored [-Wattributes] grep.c:1834: error: 'optind' redeclared without dllimport attribute after being referenced with dll linkage * configure.ac (GNULIB_WARN_FLAGS): Disable -Wunused-macros. * src/grep.c (print_line_middle): Use correct type to tolower. (main): Drop useless redeclarations. * .gitignore: Ignore more built files. 2010-03-20 Jim Meyering tests: ensure that all programs handle [b-a] consistently * tests/reversed-range-endpoints: New test. * tests/Makefile.am (TESTS): Add it. 2010-03-20 Jim Meyering build: update gnulib submodule to latest This pulls in the latest regex module from gnulib, including a fix to make it honor the RE_NO_EMPTY_RANGES syntax bit. tests: temporarily disable irrelevant-to-grep failing C++ fcntl-h-tests * bootstrap.conf (gnulib_tool_option_extras): Temporarily add --avoid=fcntl-h-tests, until the C++ part of that test is fixed. 2010-03-20 Jim Meyering reject reversed-endpoint ranges, with all regex variants * src/search.c: Add RE_NO_EMPTY_RANGES to the syntax bits in three places, so that all of grep, egrep, and grep -E reject a range with reversed endpoints like '[b-a]'. This is required, when using the latest version of gnulib's regex module, since it now honors the RE_NO_EMPTY_RANGES flag, rather than acting as if it were always set. Based on a change by Matthew Burgess. 2010-03-19 Jim Meyering maint: correct macro parameter parentheses * src/dfa.c (FETCH_WC, FETCH): Parenthesize macro parameters. 2010-03-19 Paolo Bonzini tests: change help-version to per-program functions * help-version: Change each *_args variable to a *_setup function. dfa: fix wchar_t/wint_t type mismatch * src/dfa.c (FETCH_WC): Pass a local wchar_t variable to mbrtowc. (FETCH): Rename temporary second argument to FETCH_WC. (parse_bracket_exp): Always use FETCH_WC. 2010-03-19 Jim Meyering doc: add README-prereq, referenced from README-hacking * README-prereq: New file. Cloned from coreutils, s/coreutils/grep/ Reported by Tony Abou-Assaleh. 2010-03-19 Arnold Robbins maint: sync dfa comments from gawk * src/dfa.h (struct dfa) [newlines]: Amend comment. * src/dfa.c: Update copyright year list to include gawk's. 2010-03-17 Jim Meyering maint: remove obsolete "cvs-clean" make target * Makefile.am (cvs-clean): Remove obsolete target. 2010-03-17 Paolo Bonzini dfa: initialize struct mbcset using memset * src/dfa.c (parse_bracket_exp): Use memset to initialize workmbc. dfa: spell out "unsigned int" * dfa.c (setbit, tstbit, clrbit, setbit_case_fold, lex, dfaoptimize, free_mbdata): Put "int" after unsigned. * dfa.h (struct position, struct dfa): Likewise. 2010-03-17 Paolo Bonzini dfa: optimize simple character sets under UTF-8 charsets Only use a bitset when possible without involving MBCSET. Testcase: yes 'the quick brown fox jumps over the lazy dog' | sed 100000q | \ time grep -c [ABCDEFGHIJKLMNOPQRSTUVWXYZ,] Before: 51ms (best of three runs); after: 16ms(best of three runs). * src/dfa.c (parse_bracket_exp): For simple bracket expressions under UTF-8, use a CSET. 2010-03-17 Paolo Bonzini dfa: speed up handling of brackets This patch has two sides. One is to fold the parsing of brackets in the single- and multi-byte cases. The second is to leverage this change, and use a bitset to test for single-byte characters in the charset. Splitting the two would be very hard. Testcase: yes 'the quick brown fox jumps over the lazy dog' | sed 100000q | \ time grep -c [ABCDEFGHIJKLMNOPQRSTUVWXYZ,] Before: 59ms (best of three runs); after: 51ms (best of three runs). Nice, but mostly providing infrastructure for the next patch. * src/dfa.c (setbit_case_fold): Try applying towlower/towupper. (looking_at): Remove. (FETCH_WC): New. (fetch_wc): Merge into FETCH_WC [MBS_SUPPORT]. (FETCH) [MBS_SUPPORT]: Call FETCH_WC. (prednames, find_pred, is_blank and other predicates): Move above, remove K&R syntax support. (parse_bracket_exp): New name of parse_bracket_exp_mb, rewritten to include single-byte character set parsing of brackets. (lex): Adjust for fetch_wc->FETCH_WC change, remove single-byte character set parsing of brackets. (match_mb_charset): Test against work_mbc->cset. * src/dfa.h (struct mb_char_classes): Add cset. 2010-03-17 Paolo Bonzini syntax-check: remove space-tab exception * .x-sc_space_tab: Remove. * src/dfa.c: Fix space-tab occurrence. THANKS: fix Jim Meyering's email address * THANKS: Jim is now with Red Hat. dfa: add missing function * src/dfa.c (using_utf8): New. (addtok_wc, free_mbdata, dfaoptimize) [!MBS_SUPPORT]: Do not define. (dfacomp) [!MBS_SUPPORT]: Do not call dfaoptimize. tests: fix typo * fedora: Fix typo. tests: use Exit * euc-mb: exit with "Exit 0". grep: remove more register keywords * dosbuf.c: Remove register keywords. * grep.c: Remove register keywords. * kwset.c: Remove register keywords. * search.c: Remove register keywords. 2010-03-17 Paolo Bonzini dfa: run simple UTF-8 regexps as a single-byte character set This provides a speedup whenever fgrep is "almost" sufficient but not quite (e.g. grep ^abc). This affects test cases such as https://savannah.gnu.org/bugs/?29117, which are already worked around by the line-by-line matching patch c32c04; without that patch the speedup can reach 1000x even on non-contrived testcases. * src/dfa.c (dfaoptimize): New. (dfacomp): Call it. 2010-03-17 Paolo Bonzini tests: fix syntax-check failures * tests/case-fold-backref: Use "foo" instead of "the". * tests/dfaexec-multibyte: Remove trailing blanks. 2010-03-17 Paolo Bonzini grep: remove check_multibyte_string, fix non-UTF8 missed match Avoid computing ahead something that can be computed lazily as efficiently (or more efficiently in the case of UTF-8, though this is left as TODO). At the same time, "soften" the rejection condition for matching in the middle of a multibyte sequence to fix bug 23814. Multibyte "grep -i" would still be very slow if it wasn't for the workaround patch c32c042 (grep: match multibyte charsets line-by-line when using -i, 2010-03-08). * NEWS: Document bugfix. * src/search.c (check_multibyte_string): Rewrite as... (is_mb_middle): ... this. (EGexecute, Fexecute): Adjust. * tests/Makefile.am (TESTS): Add euc-mb. * tests/euc-mb: New testcase. 2010-03-17 Paolo Bonzini dfa: cache MB_CUR_MAX for dfaexec * src/dfa.c (state_index, dfaexec): Use d->mb_cur_max. (dfainit): Initialize it. (free_mbdata): New, extracted out of dfafree. (dfafree): Use it. dfa: improve documentation of struct dfa * src/dfa.h (struct dfa): Reword some comments. tests: factor name of output files into a variable * tests/case-fold-backref, tests/case-fold-char-class, tests/case-fold-char-range, tests/case-fold-char-type, tests/dfaexec-multibyte: Use a variable for the output filename, as it is common to the grep and compare invocations. tests: use different output files to simplify reading failed .log files * tests/case-fold-backref, tests/case-fold-char-class, tests/case-fold-char-range, tests/case-fold-char-type: Use a different name for each output file from grep. * tests/dfaexec-multibyte: Likewise, and merge some grep invocations. tests: add another grep -i testcase, from bug 16179 * tests/case-fold-backref: New. * tests/Makefile.am (TESTS): Add it. 2010-03-16 Paolo Bonzini dfa: rewrite handling of multibyte case_fold lexing Let dfacomp do the folding to lowercase of multibyte input strings, and remove it from grep.c. Input strings to kwset.c are still folded outside kwset.c, so we still need to do mbtolower in search.c. * NEWS: Document bugfixes. * .x-sc_cast_of_argument_to_free: Remove. * src/dfa.c (wctok, addtok_wc): New. (cur_mb_index, update_mb_len_index): Remove. (FETCH): Do not call it. (parse_bracket_exp_mb) [GREP]: Disable case-folding of ranges and characters. (addtok): Extract part to... (addtok_mb): ... this new function. (lex): Call fetch_wc in the main loop for MB_CUR_MAX > 1. Return WCHAR for normal characters if MB_CUR_MAX > 1. (atom): Handle WCHAR instead of treating multibyte characters specially. Do case folding of multibyte characters here. (dfacomp): Remove case_fold special casing. * src/dfa.h (WCHAR): New. * src/grep.c (mb_icase_keys): Remove. (main): Do not call it. * src/search.c (kwsinit): Init transition table only for MB_CUR_MAX == 1. (mbtolower): New. (kwsincr_case): New. (kwsmusts): Call it instead of kwsincr. (check_multibyte_string): Remove. (check_multibyte_string_no_icase): Rename to check_multibyte_string. (GEAcompile, EGexecute, Fcompile): Use mbtolower instead of the old check_multibyte_string. * tests/Makefile.am (TESTS): Add case-fold-backslash-w. * tests/foad1.sh: Enable fixed tests. * tests/case-fold-backslash-w: New. 2010-03-16 Paolo Bonzini grep: match multibyte charsets line-by-line when using -i The turtle combination -i + MB_CUR_MAX>1 requires case conversion ahead of time. Avoid doing this repeatedly when many matches succeed. Together with the previous changes, this fixes https://savannah.gnu.org/bugs/?29117 and https://savannah.gnu.org/bugs/?14472. * NEWS: Document new speedup. * src/grep.c (do_execute): New. (grepbuf): Use it. 2010-03-15 Paolo Bonzini dfa: fix handling of ranges in multibyte character sets * src/dfa.c (parse_bracket_exp_mb): Add separate ranges for lowercase and uppercase endpoints if folding case. * tests/Makefile.am (TESTS): Add case-fold-char-range. * tests/case-fold-char-range: New. tests: add more UTF-8 test cases * tests/Makefile.am (TESTS): Add spencer1-locale. (EXTRA_DIST): Add spencer1-locale.awk. * tests/spencer1-locale.awk: New. * tests/spencer1-locale: New. 2010-03-15 Jim Meyering tests: complete the renaming fedora.sh -> fedora * tests/Makefile.am (TESTS): Rename fedora.sh -> fedora here, too. 2010-03-15 Jim Meyering * tests/fedora.sh: Rename to... * tests/fedora: ...this, to reflect new convention: Use the lack of a suffix to indicate we've converted to the new init.sh-using test framework. tests: adjust fedora.sh to handle traps more portably 2010-03-15 Jim Meyering tests: adjust fedora.sh to handle traps more portably * tests/fedora.sh: Use "Exit", not "exit". tests: for each test, set an envvar to its name * tests/Makefile.am (TESTS_ENVIRONMENT): Set GREP_TEST_NAME for each test. This is used to help make the output of hundreds of independent, often-parallel valgrind runs more manageable. 2010-03-14 Jim Meyering tests: clean up fedora.sh * tests/fedora.sh: Use "grep", not ${GREP}. Use init.sh. Use timeout 10, not sleep 1 (three times). The latter would always sleep for 3 seconds, and the test would fail with a false positive on a slow system or with a heavily instrumented (valgrind) executable. 2010-03-12 Jim Meyering build: avoid build failure with --enable-gcc-warnings * src/dfa.c: Don't include , now that it is not used. [DEBUG]: Remove #ifdef block. 2010-03-12 Paolo Bonzini syntax-check: enable space-tab * cfg.mk (local-checks-to-skip): Enable space-tab. * .x-sc_space_tab: Add exceptions. * tests/status.sh: Fix occurrence. syntax-check: enable m4-quote-check * cfg.mk (local-checks-to-skip): Enable m4-quote-check. * configure.ac: Fix occurrence. syntax-check: enable makefile-TAB-only-indentation * cfg.mk (local-checks-to-skip): Enable makefile-TAB-only-indentation. * Makefile.am: Fix only occurrence. grep: fix error-message-uppercase * cfg.mk (local-checks-to-skip): Enable error-message-uppercase. * src/dfa.c (parse_bracket_exp_mb, lex, dfaparse): Fix occurrences. * src/search.c (Pcompile, Pexecute): Fix occurrences. dfa, grep: cleanup if-before-free and cast-of-argument-to-free * .x-sc_avoid_if_before_free: Remove. * .x-sc_cast_of_alloca_return_value: Remove. * .x-sc_cast_of_x_alloc_return_value: Remove. * .x-sc_cast_of_argument_to_free: Temporarily add src/search.c. * cfg.mk (local-checks-to-skip): Remove sc_cast_of_argument_to_free. * src/dfa.c (ifree): Remove. (dfamust, build_state, transit_state, dfafree): Do not do if-before-free, do not cast free argument to ptr_t or char *. (freelist): Call free instead of ifree. * src/dfa.h (ptr_t): Remove. 2010-03-12 Paolo Bonzini dfa: remove CRANGE dead code The only use of CRANGE was removed by commit 193830d. In theory it is more correct to do what CRANGE did, but in practice it seems like it did not work. * src/dfa.h (token): Remove CRANGE. * src/dfa.c (atom): Do not handle CRANGE. (prtok): Likewise. 2010-03-12 Paolo Bonzini dfa: get rid of x*alloc * src/dfa.c: Include xalloc.h. (xmalloc, xrealloc, xcalloc): Remove. grep: cleanup one const cast * src/search.c (GEAcompile): Do not reuse motif when operating on the (const) pattern, so we can make it non-const. Remove cast from free. kwset/system: remove ptr_t * src/kwset.h: Declare kwset using an incomplete struct type. * src/system.h (ptr_t): Remove. 2010-03-12 Jim Meyering tests: add test cases for dfaexec bug * tests/dfaexec-multibyte: New test. * tests/Makefile.am (TESTS): Add it. Reported by Paolo Bonzini in http://bugzilla.redhat.com/544407 and http://bugzilla.redhat.com/544406 . 2010-03-12 Jim Meyering dfa: manually merge gawk's dfaexec * src/dfa.c (dfaexec): Adjust API: return pointer, not offset, and take an "end" pointer parameter, rather than integral "size". Adjust comment accordingly. (build_state): Maintain d->newlines. (copytoks): Update multibyte_prop indices. (SKIP_REMAINS_MB_IF_INITIAL_STATE): Update a cast. Return NULL, rather than (size_t) -1. (realloc_trans_if_necessary): Realloc d->newlines. * src/dfa.h (struct dfa): New member, "newlines". (struct dfa) [GAWK]: New member, "broken". (dfaexec): Update prototype and copy the new comment from dfa.c. dfa: make search.c use the new dfaexec API * src/search.c: Adjust to new dfaexec API. Now, dfaexec returns a pointer, not an integer, and the third parameter is END, not buffer size. * src/dfa.c (dfaexec): Rewrite the function's comment. Don't just clobber *END. While doing that happens to be fine for gawk's usage, in grep, *END usually points to the first byte of the next buffer. Save the initial value, and restore it just before returning. * src/dfa.h (dfaexec): Update comment; include parameter names. 2010-03-12 Jim Meyering dfa: appease static analyzers * src/dfa.c (transit_state_singlebyte): Call abort rather than returning in a "can't happen" scenario. This stops clang from emitting a false-positive report (I think it was used-uninitialized) about a caller. 2010-03-11 Jim Meyering dfa: do not accept [[:UPPER:]] or [[:LOWER:]] internally * src/dfa.c (parse_bracket_exp_mb): Those class names are not valid, and rejected elsewhere, so there is no point in allowing upper or mixed-case versions here. 2010-03-11 Jim Meyering maint: remove a trailing space * src/search.c (EXECUTE_FCT): Remove trailing space. maint: remove all uses of PARAMS Remove most with this: git grep -lw PARAMS |xargs perl -pi -e 's/\bPARAMS *\((.*)\);/$1;/' Remove the remainder manually. 2010-03-11 Jim Meyering maint: remove all uses of PARAMS * lib/savedir.h (PARAMS): Remove definitions manually. Remove the remaining ones via this command: git grep -l define.PARAMS |xargs perl -ni -e '/define PARAMS/ or print' * src/dfa.h (PARAMS): Remove definitions. * src/system.h (PARAMS): Likewise. Remove most uses with this: git grep -lw PARAMS |xargs perl -pi -e 's/\bPARAMS *\((.*)\);/$1;/' Remove the remainder manually. maint: remove now-useless prototypes * src/dfa.c: Remove the prototype of each static, non-recursive function whose definition precedes first use. grep: plug an inconsequential leak * src/grep.c (main): Plug a leak: free "keys". grep: avoid useless allocations for empty GREP_OPTIONS * src/grep.c (prepend_default_options): Ignore GREP_OPTIONS when it's empty, not just when it's undefined. There are still relatively harmless leaks when GREP_OPTIONS is set and non-empty. We'll address those, eventually. 2010-03-09 Jim Meyering build: record build-from-clone tool requirements * bootstrap.conf (buildreq): This makes bootstrap fail with a clear explanation of the problem. Otherwise, you'd get into the build process and fail with something far more cryptic. dfa: remove a trailing blank * src/dfa.c (dfaexec): No trailing blanks allowed. dfa: sync a tiny change from gawk * src/dfa.c (state_index) [MBS_SUPPORT]: Initialize .mpbs.nelem member unconditionally. Also initialize .mbps.elems. dfa: avoid a leak (work_mbc->chars) * src/dfa.c (parse_bracket_exp_mb): Remove useless (and leaked MALLOC). doc+bootstrap: document build-from-git-clone process * bootstrap: Update from coreutils/gnulib. * README-hacking: New file, nearly identical to the one in coreutils. 2010-03-08 Paolo Bonzini more work on TODO * TODO: More work on the first section. Use clearer section headers. 2010-03-08 Reuben Thomas bring TODO up-to-date * TODO: merge with TODO section of http://www.gnu.org/software/grep/devel.html and remove done items. Some small bits of tidying also. 2010-03-07 Paolo Bonzini simplify parsing of [a-z] * src/dfa.c (in_coll_range): New. (lex): Use it instead of regcomp/regexec. Small refactoring in src/dfa.c * src/dfa.c (parse_bracket_exp_mb): Return MBCSET. (lex): Assign return value of parse_bracket_exp_mb to lasttok, return it. use do...while(0) idiom * dfa.c (FETCH): Wrap with do...while(0). 2010-03-06 Paolo Bonzini extract common code from if/else * dfa.c (dfaexec): Simplify logic for MB_CUR_MAX > 1 case. remove register variable hacks * dfa.c (dfaexec): We can extract the address of a variable without fearing performance problems, modern compilers know better. remove register keywords * dfa.c (dfaexec): Modern compilers just ignore it. allow grep -Pz * NEWS: Document grep -P improvements. * src/search.c (Pcompile): Remove restriction on grep -Pz. * tests/pcre-z: New. * tests/Makefile.am (TESTS): Add pcre-z. fix cross-line matching in PCRE backend * search.c (Pexecute): Split the buffer in lines and match each line separately. * tests/fedora.sh: Add regression testsuite. fix formatting of NEWS * NEWS: fix formatting of 2.6 entries. fix a bug in handling of -i and character type * dfa.c (parse_bracket_exp_mb): Convert [[:lower:]] and [[:upper]] to [[:alpha:]] when folding case. * tests/case-fold-char-type: New file. Test for the bug. * tests/Makefile.am (TESTS): Add it. * NEWS (Bug fixes): Mention it. fix previous test case change * tests/case-fold-char-class: Do not reset fail to 0 after first test. 2010-03-06 Mike Frysinger grep(1) man page: touchup --label option * doc/grep.1 (--label): Don't italicize ending period. Point to -H option. 2010-03-06 Paolo Bonzini augment case-fold-char-class test case * tests/case-fold-char-class: Test matching lowercase against uppercase as well as vice versa. 2010-03-05 Reuben Thomas doc: improve the discussion of PCRE * doc/grep.1: Add a sentence about Perl regular expressions, and point to pcresyntax(3) and pcrepattern(3). * doc/grep.texi: Likewise. 2010-03-05 Jim Meyering maint: dfa-sync: comment and dead-to-grep code: no semantic change * src/dfa.c: Sync a comment and some #ifdef GAWK code. maint: dfa-sync: don't malloc zero * src/dfa.c (dfacomp): Skip case_fold logic when length is zero. This probably "no semantic change", but does improve efficiency in a degenerate case. maint: dfa-sync: use CALLOC rather than equiv. MALLOC+initialize-loop * src/dfa.c (dfaanalyze): Sync from gawk. No semantic change. dfa.c: add support for \s and \S * src/dfa.c (lex): Sync from gawk's dfa.c. maint: dfa-sync: add omitted array initializer * src/dfa.c (prednames): Add a "0" to final initializer. No semantic change. fix a bug in handling of -i and character classes * dfa.c (parse_bracket_exp_mb): Sync one part of this function from gawk's dfa.c, which was patched by Arnold D. Robbins. * tests/case-fold-char-class: New file. Test for the bug. * tests/Makefile.am (TESTS): Add it. (TESTS_ENVIRONMENT): Propagate LOCALE_FR and LOCALE_FR_UTF8 definitions into tests. * NEWS (Bug fixes): Mention it. 2010-03-05 Paolo Bonzini Fedora Grep regression test suite * tests/Makefile.am (TESTS): Add fedora.sh. (CLEANFILES): Add several new files. * tests/fedora.sh: New file, originally by Lubomir Rintel but somewhat rewritten to avoid bashisms. 2010-03-05 Paolo Bonzini convert AUTHORS file to UTF-8 * AUTHORS: Convert to UTF-8. eliminate invalid "ptr += (ptr2 - ptr1)" * lib/savedir.c (savedir): new_name_space and name_space do not point into the same object, so computing their difference is invalid. Similarly, summing the difference to namep is invalid because namep and the result point into different objects. Avoid this. fix for bug 21276 * lib/savedir.c (isdir1): Use realloc instead of calloc. Remove dead code. (savedir): Do not leak name_space if allocation of new_name_space fails. 2010-03-04 Jim Meyering tests: add a test based on an example from Paolo Bonzini * tests/word-multi-file: New test. * tests/Makefile.am (TESTS): Add it. doc: document release procedure * README-release: New file. build: update gnulib submodule to latest 2010-02-22 Paolo Bonzini add --group-separator=FOO and --no-group-separator * src/grep.c (group_separator): New. (long_options): Add --group-separator=FOO and --no-group-separator. (prtext): Print group_separator instead of SEP_STR_GROUP. Optionally suppress the separator altogether. (main) Handle GROUP_SEPARATOR_OPTION. * doc/grep.texi (Context control): Document it. * NEWS: Mention it. * tests/yesno.sh: Add testcases. 2010-02-21 Jim Meyering tests: don't use "echo -n" * tests/foad1.sh: Use printf, not echo -n. The latter is not portable. Reported by Daniel Richman. 2010-02-08 Jim Meyering remove useless DJGPP-specific code * src/grep.c (grepfile): Remove now-useless DJGPP-specific code. Now, all S_IS* macros are guaranteed to be defined via gnulib. 2010-02-07 Jim Meyering tests: add help-version sanity tests from coreutils * tests/help-version: New test, from coreutils. * tests/Makefile.am (TESTS): Add it. (TESTS_ENVIRONMENT) [built_programs]: Define it. tests: correct TESTS_ENVIRONMENT's PATH setting * tests/Makefile.am (TESTS_ENVIRONMENT): Set PATH to start with $(abs_top_builddir)/src, so that we test the programs we've just built. grep: use the correct exit status (2) upon write failure, not 1 * src/grep.c (main): Initialize exit_failure to EXIT_TROUBLE. * NEWS (Bug fixes): Mention this fix. maint: enable the prohibit_magic_number_exit syntax check * cfg.mk (local-checks-to-skip): Remove sc_prohibit_magic_number_exit, to enable that check. * src/system.h (EXIT_TROUBLE): Define. * src/grep.c: Use symbolic names, EXIT_SUCCESS, EXIT_FAILURE, and EXIT_TROUBLE, not 0, 1, 2. * src/search.c: Likewise. * src/vms_fab.c (string): Likewise. 2010-02-04 Jim Meyering doc: adjust NEWS item * NEWS: Correct a description. 2010-02-03 Jim Meyering tests: exercise surprising -m1 vs. --context behavior * tests/max-count-vs-context: New test. Exercise the surprising, but documented, behavior reported by Markus Jochim in http://savannah.gnu.org/bugs/?28588. * tests/Makefile.am (TESTS): Add it. tests: use init.sh from gnulib * tests/init.sh: New file, from gnulib. * tests/Makefile.am (EXTRA_DIST): Add it. (TESTS_ENVIRONMENT): Add variables and features. (VERBOSE): Define. maint: remove unused Makefile rule * tests/Makefile.am (dist-hook): Remove rule. No longer needed. maint: adjust formatting in tests/Makefile.am * tests/Makefile.am (TESTS, CLEANFILES): Align and sort. build: avoid warnings in gnulib-supplied regex files Now that we enable more warnings in lib/, we choose to avoid some via patches applied by bootstrap, using files in the gl/ hierarchy. Other, less-important warnings are avoided simply by turning off the -Wold-style-definition option and using a slightly relaxed set of warnings $(GNULIB_WARN_CFLAGS) in lib/. * gl/lib/regcomp.c.diff: Avoid warnings. * gl/lib/regex_internal.c.diff: Likewise. * gl/lib/regex_internal.h.diff: Likewise. * gl/lib/regexec.c.diff: Likewise. * configure.ac (GNULIB_PORTCHECK): Disable only -Wold-style-definition. * lib/Makefile.am (AM_CFLAGS): Use $(GNULIB_WARN_CFLAGS) rather than the slightly more strict $(WARN_CFLAGS). tests: adjust spencer #37 to pass with gnulib's regex code * tests/spencer1.tests: Change #37 to expect an exit status of 2, not 1. grep 'a[b-a]' reports "Invalid range end". maint: use regex from gnulib, rather than our bit-rotting one * bootstrap.conf (gnulib_modules): Add regex. * configure.ac: Don't use jm_INCLUDED_REGEX. Update use of cache variable. * lib/regex.c: Remove file. * lib/regex.h: Likewise. * m4/regex.m4: Likewise. * POTFILES.in: Update to match. build: update gnulib submodule to latest 2010-01-28 Jim Meyering maint: update to latest gnulib; adjust cfg.mk * gnulib: Update submodule to latest. * cfg.mk (old_NEWS_hash): Update to reflect NEWS Copyright line change. 2010-01-06 Jim Meyering maint: avoid old jm_* macros There were jm_* macros here, until very recently. * cfg.mk (sc_prohibit_jm_in_m4): New rule, from coreutils. maint: remove decl.m4 * m4/decl.m4: Remove unused file. maint: rely on gnulib's new isdir.h * src/grep.c: Include "isdir.h". * src/system.h: Remove declaration of isdir. build: rename local to avoid shadowing global, dfa * src/dfa.c (dfamust): Rename parameter: s/dfa/d/. build: avoid warning from -Wmissing-prototypes * src/dfa.c (match_mb_charset): Declare to be static. build: avoid shadowing warning for "link" * src/kwset.c (link): Define to kwset_link, to avoid shadowing the function. build: avoid shadowing warning for unused "rs" * src/dfa.c (transit_state): Remove dead stores; move a declaration "down". Ignore transit_state_consume_1char return value. build: avoid shadowing warnings * src/dfa.c (match_mb_charset): Rename parameter: s/index/idx/. (check_matching_with_multibyte_ops, match_anychar): Likewise. build: avoid warning about unused definition of N_ * src/dfa.c (N_): Remove unused definition. build: avoid format-string warnings * src/search.c (dfaerror): Use literal "%s" as format string. (kwsmusts, GEAcompile): Likewise. (Pcompile): Likewise. build: add configure-time --enable-gcc-warnings option; avoid warnings * bootstrap.conf (gnulib_modules): Add "manywarnings" module. * configure.ac: Add --enable-gcc-warnings, derived from code in bison. * src/Makefile.am (AM_CFLAGS): Set to $(WARN_CFLAGS) $(WERROR_CFLAGS) * lib/Makefile.am (AM_CFLAGS): Likewise, but append. build: remove now-useless -I../intl option * src/Makefile.am (INCLUDES): Remove -I../intl, now that intl is gone. maint: avoid more warnings * src/grep.c (MAX): Remove definition of unused macro. (usage): Declare with __attribute__ ((noreturn)). Split long strings into chunks of length < 509. fix a possible bug: remove errant semicolon * src/grep.c (prline): Remove erroneous semicolon-after-if-expr. maint: avoid compilation warnings * bootstrap.conf (gnulib_modules): Add ignore-value. * src/search.c (check_multibyte_string_no_icase): A variant of check_multibyte_string that does *not* convert case, and hence does not modify its BUF parameter. (check_multibyte_string): Use xcalloc in place of xmalloc+memset. Use ignore_value to ignore the return value from wcrtomb. This is ok, since we know the input is a valid upper case wide character. (Fexecute, EGexecute): Update callers of check_multibyte_string to use both it and check_multibyte_string_no_icase. maint: avoid warnings about unused fwrite return value * bootstrap.conf (gnulib_modules): Add unlocked-io. * src/system.h: Include "unlocked-io.h". maint: remove {m4,lib}/.gitignore; they were undergoing too much churn * .gitignore: Ignore all of m4/* except m4/djgpp.m4 and all of lib/* except Makefile.am, savedir.c and savedir.h. * m4/.gitignore: Remove file. * lib/.gitignore: Remove file. 2010-01-05 Jim Meyering build: run gnulib's tests, too * Makefile.am (SUBDIRS): Add gnulib-tests. * gnulib-tests/Makefile.am: New file. * bootstrap.conf (bootstrap_epilogue): New function, from coreutils. (gnulib_tool_option_extras): Define. * configure.ac: Add gnulib-tests/Makefile. 2010-01-03 Jim Meyering maint: record update-copyright options for this package * cfg.mk: Next time, just run "make update-copyright". 2010-01-01 Jim Meyering maint: update all FSF copyright year lists to include 2010 Use this command: git ls-files |grep -vE '^(\..*|COPYING|gnulib)$' |xargs \ env UPDATE_COPYRIGHT_USE_INTERVALS=1 build-aux/update-copyright 2009-12-23 Jim Meyering fix multi-byte-locale read-beyond-end-of-buffer error Avoid read-beyond-end-of-buffer errors, evoked by running this: LC_ALL=en_US.UTF-8 valgrind src/grep -f <(printf 'a\nb\n') <(echo c) Conditional jump or move depends on uninitialised value(s) at 0x78136D: __gconv_transform_utf8_internal (in /lib/libc-2.11.so) by 0x7E7232: mbrtowc (in /lib/libc-2.11.so) by 0x8055773: dfaexec (dfa.c:2816) by 0x804D7B0: EGexecute (search.c:353) by 0x804ACD8: grepbuf (grep.c:1036) by 0x804B023: grep (grep.c:1156) by 0x804B460: grepfile (grep.c:1287) by 0x804CF0D: main (grep.c:2282) Conditional jump or move depends on uninitialised value(s) at 0x7E7248: mbrtowc (in /lib/libc-2.11.so) by 0x8055773: dfaexec (dfa.c:2816) by 0x804D7B0: EGexecute (search.c:353) by 0x804ACD8: grepbuf (grep.c:1036) by 0x804B023: grep (grep.c:1156) by 0x804B460: grepfile (grep.c:1287) by 0x804CF0D: main (grep.c:2282) * src/dfa.c (dfaexec) [MBS_SUPPORT]: Do not access one byte beyond end of buffer. 2009-12-23 Jim Meyering build: update gnulib submodule to latest 2009-12-23 Paolo Bonzini Speed up insert. Suggested by Johan Walles (bug 23354). * src/dfa.c (insert): Use binary search. 2009-12-23 Johan Walles Decrease epsclosure memory usage Fixes bug 23321. * src/dfa.c (epsclosure): Make visited an array of char. 2009-12-22 Paolo Bonzini Make 'grep -1 -2' and 'grep -1v2' equivalent to grep -2 Fixes bug 12128. * src/grep.c (get_nondigit_option): Reset the buffer every time a non-digit option is found or a new argument is started. 2009-12-22 Paolo Bonzini Improve description of --label Fixes bug 22681. * doc/grep.1 (--label): Use -H in the example, improve wording. * doc/grep.texi (Output Line Prefix Control): Likewise. 2009-12-22 Paolo Bonzini Avoid using an invalid memchr result. Related to bug 13161. I cannot find a testcase, but it is better to be defensive considering that these bug were found in the past. * src/search.c (EGexecute, Fexecute): Check for memchr return values. 2009-12-11 Jim Meyering build: update gnulib submodule to latest 2009-12-04 Jim Meyering maint: enable prohibit_have_config_h check * cfg.mk (local-checks-to-skip): Enable sc_prohibit_have_config_h * lib/regex.c: Remove useless cpp test of HAVE_CONFIG_H. * lib/savedir.c: Likewise. * src/grep.c: Likewise. * src/kwset.c: Likewise. * src/search.c: Likewise. maint: enable cast_of_x_alloc_return_value check * cfg.mk (local-checks-to-skip): Enable sc_cast_of_x_alloc_return_value. * .x-sc_cast_of_x_alloc_return_value: * src/dfa.c (CALLOC, MALLOC, REALLOC): Remove casts. * src/dosbuf.c (undossify_input): Likewise. * src/grep.c (print_line_middle, prepend_default_options): Likewise. maint: enable cast_of_alloca_return_value check * cfg.mk (local-checks-to-skip): Enable sc_cast_of_alloca_return_value. * .x-sc_cast_of_alloca_return_value: New file. 2009-12-04 Paolo Bonzini fix "grep -Ff" on CRLF-terminated files * src/search.c (Fcompile) [HAVE_DOS_FILE_CONTENTS]: Recognize \r\n as a line terminator. fix compilation with included regex * Makefile.am (libgreputils_a_DEPENDENCIES): New. switch to pkg-config for PCRE detection * configure.ac: use pkg-config to detect PCRE * src/Makefile.am (grep_LDADD): link grep with PCRE_LIBS 2009-12-04 Jim Meyering maint: remove "missing" script * missing: Remove now-unused file. maint: make .gitignore ignore more * .gitignore: Ignore more. maint: enable useless-if-before-free check * cfg.mk (local-checks-to-skip): Enable sc_avoid_if_before_free. * .x-sc_avoid_if_before_free: New file. Exempt regex.c and dfa.c, in case anyone ever tries to merge their contents with other versions. * src/grep.c (print_line_middle, grepdir): Remove useless if-before-free. * src/search.c (IF_BK, EXECUTE_FCT): Likewise. maint: enable po-check * cfg.mk (local-checks-to-skip): Enable sc_po_check. * po/POTFILES.in: Sort and update. 2009-12-03 Paolo Bonzini update gnulib, fixing missing inclusion of stdbool.h * gnulib: Update. 2009-11-30 Jim Meyering maint: enable two checks * cfg.mk (local-checks-to-skip): Enable two: sc_prohibit_xalloc_without_use sc_two_space_separator_in_usage * src/grep.c (usage): Conform: use two spaces, not 1. * src/kwset.c (malloc): Define as a function-macro so that the syntax-check rule sees that we are indeed using xmalloc here. maint: enable makefile_path_separator check * cfg.mk (local-checks-to-skip): Enable sc_makefile_path_separator_check, now that the sole offender, an old po/Makefile.in.in, is gone. maint: remove now-generated file: po/Makefile.in.in * po/Makefile.in.in: Remove file, now generated via bootstrap. maint: enable makefile @...@ check * cfg.mk (local-checks-to-skip): Enable sc_makefile_check. * lib/Makefile.am (libgreputils_a_LIBADD): Use $(...), rather than anachronistic @...@ notation. * src/Makefile.am (LDADD): Likewise. * tests/Makefile.am (AWK): Remove definition. maint: enable trailing_blank check * cfg.mk (local-checks-to-skip): Enable sc_trailing_blank. * AUTHORS: Remove trailing blanks. * COPYING: Likewise. * README: Likewise. * README-alpha: Likewise. * README-boot: Likewise. * THANKS: Likewise. * TODO: Likewise. * src/dfa.c: Likewise. * src/mbsupport.h: Likewise. * tests/backref.sh: Likewise. * tests/file.sh: Likewise. * tests/options.sh: Likewise. * tests/tests: Likewise. * vms/README: Likewise. * vms/make.com: Likewise. maint: enable unmarked_diagnostics check * cfg.mk (local-checks-to-skip): Enable sc_unmarked_diagnostics * src/grep.c (fillbuf): Mark a diagnostic for translation. (reset): Likewise. maint: enable require_config_h checks * cfg.mk (local-checks-to-skip): Enable sc_require_config_h and sc_require_config_h_first. * src/dosbuf.c: Include . * src/vms_fab.c: Likewise. * .x-sc_require_config_h: New file: list the exceptions. * .x-sc_require_config_h_first: Likewise. maint: use gnulib's progname module; enable set_program_name check * bootstrap.conf (gnulib_modules): Add progname. * src/grep.c: Include "progname.h". (program_name): Remove declaration. (main): Call set_program_name. * cfg.mk (local-checks-to-skip): Add sc_program_name. maint: enable "file system" check * cfg.mk (local-checks-to-skip): Enable sc_file_system. * lib/savedir.c (savedir): Tweak spelling. Remove trailing blanks. maint: enable immutable_NEWS check * NEWS: Move copyright to the bottom. Use the format required by release-related tools. * .prev-version: New file. * cfg.mk (old_NEWS_hash): Define. (local-checks-to-skip): Enable check: sc_immutable_NEWS. maint: disable the many failing syntax-checks * cfg.mk: New file. (local-checks-to-skip): Define to the list of disabled rules. Subsequent change-sets will enable them, one by one. build: require automake-1.11, enable silent-rules, parallel tests, xz * configure.ac (AM_INIT_AUTOMAKE): Create xz-compressed tarballs, not bzip2-compressed ones. Enable automake's silent-rules, parallel tests, and test PASS/FAIL coloring options. Use AC_CONFIG_HEADERS, not AM_CONFIG_HEADER. Quote the argument. build: use git-version-gen for inter-release version strings * configure.ac (AC_INIT): Use git-version-gen. build: add several build- and release-related gnulib modules * bootstrap.conf (gnulib_modules): Add announce-gen update-copyright do-release-commit-and-tag git-version-gen gnu-web-doc-update gnupload maintainer-makefile useless-if-before-free build: adapt to the newer closeout module from gnulib * src/grep.c: Include "exitfail.h". (main) [-q]: Set the global variable, exit_failure, rather than calling the now-removed close_stdout_set_file_name function. build: adapt to the newer exclude API we now get from gnulib * src/grep.c (main): Adapt to newer exclude.c: add EXCLUDE_WILDCARDS as the new "option" argument in calls to add_exclude and add_exclude_file. build: get more lib/* files from gnulib, adjust savedir * bootstrap.conf (gnulib_modules): Add the following: closeout exclude hard-locale isdir strtoumax. * lib/.gitignore, m4/.gitignore: Update. * lib/closeout.c, lib/closeout.h: Remove. * lib/exclude.c, lib/exclude.h: Remove. * lib/hard-locale.c, lib/hard-locale.h: Remove. * lib/strtoumax.c: Remove. * lib/isdir.c: Remove. * lib/Makefile.am: Remove here, too. * lib/savedir.c: Adapt to new exclude module: s/excluded_filename/excluded_file_name/ and remove 3rd argument. build: update gnulib submodule to latest maint: generate ChangeLog from git logs * Makefile.am (dist-hook, gen-ChangeLog): New rules. * bootstrap.conf (gnulib_modules): Add gitlog-to-changelog. Ensure that ChangeLog exists. * ChangeLog-2009: Rename from ChangeLog * ChangeLog: Remove file. * .gitignore: Add ChangeLog. maint: list gnulib modules one per line * bootstrap.conf (gnulib_modules): List them one per line. 2009-11-29 Tony Abou-Assaleh Acknowledge new maintainers, update README-alpha * AUTHORS: new maintainers added * THANKS: same * README-alpha: change CVS references to Git