diff --git a/.github/workflows/build-linux.yml b/.github/workflows/build-linux.yml index b736efd5f1..227043b1ed 100644 --- a/.github/workflows/build-linux.yml +++ b/.github/workflows/build-linux.yml @@ -13,6 +13,7 @@ on: jobs: build-linux: strategy: + fail-fast: false matrix: ocaml-compiler: # Don't include every versions. OCaml-CI already covers that @@ -41,6 +42,9 @@ jobs: opam exec -- dune subst opam exec -- dune build -p ocamlformat-lib,ocamlformat + - name: Self-formatting test + run: opam exec -- dune build @fmt + - name: Runtest run: opam exec -- dune runtest diff --git a/HACKING.jst.md b/HACKING.jst.md new file mode 100644 index 0000000000..204c1aa933 --- /dev/null +++ b/HACKING.jst.md @@ -0,0 +1,204 @@ +Jane Street `ocamlformat` instructions +====================================== + +This file describes the shape of the `ocamlformat` repo, with a +specific eye toward making it easy to update `ocamlformat` to deal +with Jane Street extensions. + +Overall structure +----------------- + +The main implementation of `ocamlformat` lives in the `lib` directory. +This is where the `ocamlformat` executable is driven. + +The most important file for our purposes here is `lib/Fmt_ast.ml`, which +defines the functions that actually print the formatted AST. Other modules +of interest in `lib/`: + +* `Fmt.ml` has combinators useful for printing. You're likely to use this + module but not super likely to change it. + +* `Ast.ml` has various helpers, mainly for (a) constructing contexts, and (b) + analyzing whether terms need parentheses. It's common to need to change the + latter for new syntax. + +* `Cmts.ml` deals with comments (not the `cmt` file format!). Comments are not + places in the extended AST, but rather maintained in a table that is checked + at various points using the helpers in this module. + +* `Sugar.ml` has various bits of support code for sugaring/desugaring syntax + (e.g., "multi-argument" functions). + +`ocamlformat` also includes *four* copies of the OCaml parser, all in +`vendor`: + +* `parser-upstream` is just a copy of the upstream parser. It does not get + built in this repo. Instead, it lives here only so that it can be diffed + with `parser-standard`. This diff is automatically produced in + `vendor/diff-parsers-upstream-std.patch`. The diff is used to monitor the + drift between the standard parser and the upstream parser; the patch file + has been mangled (it is produced by `tools/diff-ocaml` which uses `sed` to + alter the output of `diff`) and thus cannot be applied by any known tool. + + You will not have to interact with `parser-upstream`. + +* `parser-standard` is meant to be very similar to the parser used in the + OCaml compiler. Its role in `ocamlformat` is (only) to provide a safety-check: + we want to make sure that the formatting does not change the AST as parsed by + the compiler. So after formatting, the parsed AST is checked against the + original parsed AST to make sure they are the same (modulo some normalization, + as written in `lib/Normalize_std_ast.ml`). Key point: this parser and its + representation are *not* pretty-printed and never interact with the + pretty-printer. The *only* reason this is here is to mimic the behavior of the + OCaml compiler. Accordingly, it should be as close to the OCaml compiler's + parser as possible. + +* `parser-extended` uses an extended parsetree, capable of storing extra + information useful in preserving the structure of the user's code. It is this + extended parsetree that gets pretty-printed. The parser here forms part of the + core implementation of `ocamlformat`. + +* `parser-recovery` is used for partial parsing, so that a bogus input source + file does not get mangled. It was an experiment that has been discontinued by + upstream and is not used with Jane Street. It uses the same parsetree as + `parser-extended`. A patchfile tracking the changes between `parser-extended` + and `parser-recovery` is generated. Just accept any changes that accrue there. + +The directory `vendor/ocaml-common` contains files that are shared between +`parser-standard` and `parser-extended`, like `location.ml`. The `test` +directory contains tests (see the Testing section below). + +Design considerations +--------------------- + +Because the value of `parser-standard` is entirely in its ability to mimic the +compiler's parser, we want to keep this parser as close as possible to the +compiler's. We will want to copy over any changes made to the compiler's parser +into this version of `ocamlformat`'s parser. + +On the other hand, the `parser-extended` can go off in its own direction: its +parsetree should be designed to make pretty-printing easy. In addition, we want +to make sure that incorporating upstream changes is easy. We thus feel free to +edit the parsetree in `parser-extended`, but we do so by adding new +constructors, not modifying existing ones. In addition, new code should be +marked off with comments like `(* Jane Street extension *)` and `(* End Jane +Street extension *)`. Because of the ability to extend the parsetree here, we do +*not* use jane-syntax in `parser-extended`. + +`ocamlformat` routinely checks, at various places in `Fmt_ast`, that the thing +it's about to print is an exact subterm of a "context" we're within (some larger +term that we're in the middle of printing). Make sure you've designed your +additions to `parser-extended` so that, while printing, you'll always recur on +exact structural subterms. If you, for example, print an attribute specially +and then try to recur on a term without that attribute, you're in for a bad +time. + +Before building +--------------- + +You will need to install several libraries. This command may work: + +``` +opam install menhir.20210419 fix ocp-indent bechamel-js alcotest campl-streams fpath either dune-build-info uuseg ocaml-version +``` + +Building +-------- + +To build, run `dune build`. + +How to update `ocamlformat` +--------------------------- + +The base branch to work from is called `jane`. Create a branch off of `jane`. + +1. Take the patch you wish to support (i.e. some PR in `flambda-backend`). + Apply any changes to the `ocaml/parsing` directory to the files in + `vendor/parser-standard`. Remember: this "standard" parser should be as + close as possible to the compiler's. + + Note that some files used by both parsers are stored in + `vendor/ocaml-common` and may need to be updated. Further, when + incorporating new support files from the compiler, consider whether than can + be shared in that directory rather than copied into each of the parser + directories. This is typically the case if the support module doesn't depend + on the parsetree. + +2. Get `ocamlformat` compiled and passing the tests. If the patch to + `flambda-backend` was backward compatible, then this should be + straightforward. (If your changes affect files in `vendor/ocaml-common`, this + might not be so easy. That's OK. Just move on to the next step.) + +3. Edit the parsetree in `vendor/parser-extended/parsetree.mli` to support your + new syntax. Copy over any changes to the parser and lexer from the + `flambda-backend` patch, updating the parser's semantic actions as necessary. + +4. Edit the pretty-printer in `lib/Fmt_ast.ml` to format your new syntax nicely. + This may require changes to other `lib/` modules, such as `Ast.ml` and + `Sugar.ml`. + +5. Make the minimal changes to `parser-recovery` in order to get `ocamlformat` + to compile. We do not use this feature within Jane Street (and it will be + removed when merging with upstream), and so we're just keeping it on life + support. Expend no extra effort here! + +6. Add tests. Get them to pass. See the "Testing" section below. + +Testing +------- + +To just run your built ocamlformat against a file manually, run +`dune exec ocamlformat -- --enable-outside-detected-project path/to/your/file.ml`. + +If you want to see the parsed source ocamlformat produces internally, use +`dune exec -- tools/printast/printast.exe path/to/your/file.ml`. + +Run the tests with `dune test`. There are two kinds of tests: + +1) Correctly formatted files, which ocamlformat is run on to check that there + are no changes. We have historically mainly added these, but not for any + particularly good reason. +2) Incorrectly formatted files, for which the output of ocamlformat is checked + against a reference. + +To add a test, you add one, two or three files depending on what kind of test it +is: + +- (Always) Add `tests/passing/tests/foo.ml` (where foo is the name of your new + test). This is the file ocamlformat will be run on. +- (Optional) If your file is incorrectly formatted, write the correctly + formatted version in `tests/passing/tests/foo.ml.ref`. +- (Optional) If it is expected `ocamlformat` will print information to stderr + when running your test (uncommon) write that output to + `tests/passing/tests/foo.ml.err`. + +Now, run `dune test`. It will discover your new file and suggest edits to +the generated `tests/passing/dune.inc` file to run your new tests. Run +`dune promote` to update `dune.inc`. This will *not* accept your new tests -- it +just allows you to run your new tests. + +Then, run `dune test` again to actually run your tests. You will see any changes +necessary to make the tests pass. You can run `dune promote` to accept those +changes. + +If you get some cryptic error output with a few lines of the `dune.inc` file, it +is likely that ocamlformat has crashed (e.g. with a parser error) while looking +at your test. The cryptic error output will mention the name of the test. Run +ocamlformat manually on your test file to see the actual error. + +Validity checking +----------------- + +The ocamlformat repo has (at least) two validity checks for repo health: + +* The ocamlformat sources themselves must be formatted. You can run this check +with `make fmt` (which will also auto-format `dune-project`). To reformat files +that are incorrect, run `dune build @fmt --auto-promote`. Running `make test` +runs `make fmt` and `dune runtest` together. The CI will check both the +formatting check and the ocamlformat tests (but will not update `dune-project`). + +* All commits must be signed off. This is easy. When you're done with your +sequence of commits and it's all ready to merge, just run +`git rebase --signoff`, where `` is the +commit before any of your edits. You can often say something like `origin/jane` +or `HEAD~4` or similar. diff --git a/doc/manpage_ocamlformat.mld b/doc/manpage_ocamlformat.mld index 2977fddff3..78cccff002 100644 --- a/doc/manpage_ocamlformat.mld +++ b/doc/manpage_ocamlformat.mld @@ -549,6 +549,12 @@ OPTIONS (or in $HOME/.config/.ocamlformat if $XDG_CONFIG_HOME is undefined) is applied. + --erase-jane-syntax + Erase all erasable Jane Street syntax extensions. Jane Street uses + this to generate the upstream-compatible public release code for + our libraries (vs. the variant with Jane Street-specific syntax). + THIS OPTION WILL CHANGE THE RESULTING AST. + -g, --debug Generate debugging output. The flag is unset by default. @@ -597,6 +603,9 @@ OPTIONS --no-quiet Unset quiet. + --no-rewrite-old-style-jane-street-local-annotations + Unset rewrite-old-style-jane-street-local-annotations. + --no-version-check Unset version-check. @@ -631,6 +640,12 @@ OPTIONS --repl-file Parse input as toplevel phrases with their output. + --rewrite-old-style-jane-street-local-annotations + Rewrite all Jane Street annotations for use with the local mode, + such as "[%local]" or "[@ocaml.global]", into their pretty-printed + syntactic form, such as "local_" or "global_". THIS OPTION WILL + CHANGE THE RESULTING AST. The flag is unset by default. + --root=DIR Root of the project. If specified, only take into account .ocamlformat configuration files inside DIR and its diff --git a/lib/Ast.ml b/lib/Ast.ml index c47beb25f2..5a68755d76 100644 --- a/lib/Ast.ml +++ b/lib/Ast.ml @@ -146,7 +146,14 @@ module Exp = struct false | _ -> List.exists pexp_attributes ~f:(Fn.non Attr.is_doc) + let maybe_extension exp f_extension f_normal = + match Extensions.Expression.of_ast exp with + | Some eexp -> f_extension eexp + | None -> f_normal () + let rec is_trivial exp = + maybe_extension exp is_trivial_extension + @@ fun () -> match exp.pexp_desc with | Pexp_constant {pconst_desc= Pconst_string (_, _, None); _} -> true | Pexp_constant _ | Pexp_field _ | Pexp_ident _ | Pexp_send _ -> true @@ -160,6 +167,14 @@ module Exp = struct | Pexp_array [x] | Pexp_list [x] -> is_trivial x | _ -> false + and is_trivial_extension : Extensions.Expression.t -> _ = function + | Eexp_immutable_array (Iaexp_immutable_array []) -> true + | Eexp_immutable_array (Iaexp_immutable_array [x]) -> is_trivial x + | Eexp_immutable_array (Iaexp_immutable_array _) + |Eexp_comprehension + (Cexp_list_comprehension _ | Cexp_array_comprehension _) -> + false + let rec exposed_left e = match e.pexp_desc with | Pexp_prefix _ -> true @@ -207,7 +222,18 @@ module Pat = struct | Ppat_cons pl when List.for_all pl ~f:is_any -> true | _ -> false - let has_trailing_attributes {ppat_desc; ppat_attributes; _} = + let maybe_extension pat f_extension f_normal = + match Extensions.Pattern.of_ast pat with + | Some epat -> f_extension epat + | None -> f_normal () + + let has_trailing_attributes_extension _ppat_attributes : + Extensions.Pattern.t -> _ = function + | Epat_immutable_array (Iapat_immutable_array _) -> false + + let has_trailing_attributes ({ppat_desc; ppat_attributes; _} as pat) = + maybe_extension pat (has_trailing_attributes_extension ppat_attributes) + @@ fun () -> match ppat_desc with | Ppat_construct (_, None) |Ppat_constant _ | Ppat_any | Ppat_var _ @@ -257,7 +283,8 @@ let rec mty_is_simple x = | Pmty_signature (_ :: _) |Pmty_with (_, _ :: _ :: _) |Pmty_extension _ - |Pmty_functor (_, _) -> + |Pmty_functor (_, _) + |Pmty_strengthen _ -> false | Pmty_gen (_, t) -> mty_is_simple t | Pmty_typeof e -> mod_is_simple e @@ -770,7 +797,7 @@ module rec In_ctx : sig val sub_pat : ctx:T.t -> pattern -> pattern xt - val sub_exp : ctx:T.t -> expression -> expression xt + val sub_exp : Conf.t -> ctx:T.t -> expression -> expression xt val sub_cl : ctx:T.t -> class_expr -> class_expr xt @@ -804,7 +831,7 @@ end = struct let sub_pat ~ctx pat = check parenze_pat {ctx; ast= pat} - let sub_exp ~ctx exp = check parenze_exp {ctx; ast= exp} + let sub_exp conf ~ctx exp = check (parenze_exp conf) {ctx; ast= exp} let sub_cl ~ctx cl = {ctx; ast= cl} @@ -831,7 +858,7 @@ and Requires_sub_terms : sig val is_simple : Conf.t -> (expression In_ctx.xt -> int) -> expression In_ctx.xt -> bool - val exposed_right_exp : cls -> expression -> bool + val exposed_right_exp : Conf.t -> cls -> expression -> bool val prec_ast : T.t -> Prec.t option @@ -843,11 +870,11 @@ and Requires_sub_terms : sig val parenze_cty : class_type In_ctx.xt -> bool - val parenze_cl : class_expr In_ctx.xt -> bool + val parenze_cl : Conf.t -> class_expr In_ctx.xt -> bool val parenze_pat : pattern In_ctx.xt -> bool - val parenze_exp : expression In_ctx.xt -> bool + val parenze_exp : Conf.t -> expression In_ctx.xt -> bool val parenze_nested_exp : expression In_ctx.xt -> bool end = struct @@ -936,7 +963,8 @@ end = struct List.exists fields ~f:(function | {pof_desc= Otag (_, t1); _} -> typ == t1 | {pof_desc= Oinherit t1; _} -> typ == t1 ) ) - | Ptyp_class (_, l) -> assert (List.exists l ~f) ) + | Ptyp_class (_, l) -> assert (List.exists l ~f) + | Ptyp_constr_unboxed (_, t1N) -> assert (List.exists t1N ~f) ) | Td {ptype_params; ptype_cstrs; ptype_kind; ptype_manifest; _} -> assert ( List.exists ptype_params ~f:fst_f @@ -1216,6 +1244,35 @@ end = struct let dump {ctx; ast= cl} = dump ctx (Cl cl) in assert_no_raise ~f:check_cl ~dump xcl + module Comprehension_child = struct + type t = Expression of expression | Pattern of pattern + end + + let check_comprehension Extensions.Comprehensions.{body; clauses} + (tgt : Comprehension_child.t) = + let expression_is_child = + match tgt with + | Expression exp -> fun exp' -> exp' == exp + | _ -> fun _ -> false + in + let pattern_is_child = + match tgt with + | Pattern pat -> fun pat' -> pat' == pat + | _ -> fun _ -> false + in + expression_is_child body + || List.exists clauses ~f:(function + | For bindings -> + List.exists bindings + ~f:(fun {iterator; pattern; attributes= _} -> + pattern_is_child pattern + || + match iterator with + | Range {start; stop; direction= _} -> + expression_is_child start || expression_is_child stop + | In seq -> expression_is_child seq ) + | When cond -> expression_is_child cond ) + let check_pat {ctx; ast= pat} = let check_extensions = function PPat (p, _) -> p == pat | _ -> false in let check_subpat ppat = @@ -1238,6 +1295,13 @@ end = struct | Td _ -> assert false | Pat ctx -> ( let f pI = pI == pat in + (* Inlined because we're closed over things *) + let check_extension : Extensions.Pattern.t -> _ = function + | Epat_immutable_array (Iapat_immutable_array p1N) -> + assert (List.exists p1N ~f) + in + Pat.maybe_extension ctx check_extension + @@ fun () -> match ctx.ppat_desc with | Ppat_array p1N | Ppat_list p1N | Ppat_tuple p1N | Ppat_cons p1N -> assert (List.exists p1N ~f) @@ -1259,33 +1323,45 @@ end = struct |Ppat_variant (_, None) -> assert false ) | Exp ctx -> ( - match ctx.pexp_desc with - | Pexp_apply _ | Pexp_array _ | Pexp_list _ | Pexp_assert _ - |Pexp_coerce _ | Pexp_constant _ | Pexp_constraint _ - |Pexp_construct _ | Pexp_field _ | Pexp_ident _ | Pexp_ifthenelse _ - |Pexp_lazy _ | Pexp_letexception _ | Pexp_letmodule _ | Pexp_new _ - |Pexp_newtype _ | Pexp_open _ | Pexp_override _ | Pexp_pack _ - |Pexp_poly _ | Pexp_record _ | Pexp_send _ | Pexp_sequence _ - |Pexp_setfield _ | Pexp_setinstvar _ | Pexp_tuple _ - |Pexp_unreachable | Pexp_variant _ | Pexp_while _ | Pexp_hole - |Pexp_beginend _ | Pexp_parens _ | Pexp_cons _ | Pexp_letopen _ - |Pexp_indexop_access _ | Pexp_prefix _ | Pexp_infix _ -> - assert false - | Pexp_extension (_, ext) -> assert (check_extensions ext) - | Pexp_object {pcstr_self; _} -> - assert (Option.exists ~f:(fun self_ -> self_ == pat) pcstr_self) - | Pexp_let ({pvbs_bindings; _}, _) -> - assert (check_bindings pvbs_bindings) - | Pexp_letop {let_; ands; _} -> - let f {pbop_pat; _} = check_subpat pbop_pat in - assert (f let_ || List.exists ~f ands) - | Pexp_function cases | Pexp_match (_, cases) | Pexp_try (_, cases) -> - assert ( - List.exists cases ~f:(function - | {pc_lhs; _} when pc_lhs == pat -> true - | _ -> false ) ) - | Pexp_for (p, _, _, _, _) | Pexp_fun (_, _, p, _) -> assert (p == pat) - ) + (* Inlined because it's simpler *) + let check_extension : Extensions.Expression.t -> _ = function + | Eexp_comprehension + ( Cexp_list_comprehension comp + | Cexp_array_comprehension (_, comp) ) -> + assert (check_comprehension comp (Pattern pat)) + | Eexp_immutable_array (Iaexp_immutable_array _) -> assert false + in + Exp.maybe_extension ctx check_extension + @@ fun () -> + match ctx.pexp_desc with + | Pexp_apply _ | Pexp_array _ | Pexp_list _ | Pexp_assert _ + |Pexp_coerce _ | Pexp_constant _ | Pexp_constraint _ + |Pexp_construct _ | Pexp_field _ | Pexp_ident _ + |Pexp_ifthenelse _ | Pexp_lazy _ | Pexp_letexception _ + |Pexp_letmodule _ | Pexp_new _ | Pexp_newtype _ | Pexp_open _ + |Pexp_override _ | Pexp_pack _ | Pexp_poly _ | Pexp_record _ + |Pexp_send _ | Pexp_sequence _ | Pexp_setfield _ + |Pexp_setinstvar _ | Pexp_tuple _ | Pexp_unreachable + |Pexp_variant _ | Pexp_while _ | Pexp_hole | Pexp_beginend _ + |Pexp_parens _ | Pexp_cons _ | Pexp_letopen _ + |Pexp_indexop_access _ | Pexp_prefix _ | Pexp_infix _ -> + assert false + | Pexp_extension (_, ext) -> assert (check_extensions ext) + | Pexp_object {pcstr_self; _} -> + assert (Option.exists ~f:(fun self_ -> self_ == pat) pcstr_self) + | Pexp_let ({pvbs_bindings; _}, _) -> + assert (check_bindings pvbs_bindings) + | Pexp_letop {let_; ands; _} -> + let f {pbop_pat; _} = check_subpat pbop_pat in + assert (f let_ || List.exists ~f ands) + | Pexp_function cases | Pexp_match (_, cases) | Pexp_try (_, cases) + -> + assert ( + List.exists cases ~f:(function + | {pc_lhs; _} when pc_lhs == pat -> true + | _ -> false ) ) + | Pexp_for (p, _, _, _, _) | Pexp_fun (_, _, p, _) -> + assert (p == pat) ) | Lb x -> assert (x.pvb_pat == pat) | Mb _ -> assert false | Md _ -> assert false @@ -1338,6 +1414,17 @@ end = struct | Exp ctx -> ( let f eI = eI == exp in let snd_f (_, eI) = eI == exp in + (* Inlined because we're closed over things *) + let check_extension : Extensions.Expression.t -> _ = function + | Eexp_comprehension + ( Cexp_list_comprehension comp + | Cexp_array_comprehension (_, comp) ) -> + assert (check_comprehension comp (Expression exp)) + | Eexp_immutable_array (Iaexp_immutable_array e1N) -> + assert (List.exists e1N ~f) + in + Exp.maybe_extension ctx check_extension + @@ fun () -> match ctx.pexp_desc with | Pexp_extension (_, ext) -> assert (check_extensions ext) | Pexp_constant _ | Pexp_ident _ | Pexp_new _ | Pexp_pack _ @@ -1486,6 +1573,8 @@ end = struct let rec is_simple (c : Conf.t) width ({ast= exp; _} as xexp) = let ctx = Exp exp in + Exp.maybe_extension exp (is_simple_extension c width xexp) + @@ fun () -> match exp.pexp_desc with | Pexp_constant _ -> Exp.is_trivial exp | Pexp_field _ | Pexp_ident _ | Pexp_send _ @@ -1493,7 +1582,7 @@ end = struct |Pexp_variant (_, None) -> true | Pexp_cons l -> - List.for_all l ~f:(fun e -> is_simple c width (sub_exp ~ctx e)) + List.for_all l ~f:(fun e -> is_simple c width (sub_exp c ~ctx e)) && fit_margin c (width xexp) | Pexp_construct (_, Some e0) | Pexp_variant (_, Some e0) -> Exp.is_trivial e0 @@ -1520,10 +1609,27 @@ end = struct && List.for_all e1N ~f:(snd >> Exp.is_trivial) && fit_margin c (width xexp) | Pexp_extension (_, PStr [{pstr_desc= Pstr_eval (e0, []); _}]) -> - is_simple c width (sub_exp ~ctx e0) + is_simple c width (sub_exp c ~ctx e0) | Pexp_extension (_, (PStr [] | PTyp _)) -> true | _ -> false + and is_simple_extension c width xexp : Extensions.Expression.t -> bool = + function + | Eexp_immutable_array (Iaexp_immutable_array e1N) -> + List.for_all e1N ~f:Exp.is_trivial && fit_margin c (width xexp) + | Eexp_comprehension + (Cexp_list_comprehension _ | Cexp_array_comprehension _) -> + false + + let prec_ctx_extension : Extensions.Expression.t -> _ = + let open Prec in + let open Assoc in + function + | Eexp_comprehension + (Cexp_list_comprehension _ | Cexp_array_comprehension _) + |Eexp_immutable_array (Iaexp_immutable_array _) -> + Some (Semi, Non) + (** [prec_ctx {ctx; ast}] is the precedence of the context of [ast] within [ctx], where [ast] is an immediate sub-term (modulo syntactic sugar) of [ctx]. Also returns whether [ast] is the left, right, or neither child @@ -1579,7 +1685,9 @@ end = struct | Ptyp_constr _ -> Some (Apply, Non) | Ptyp_any | Ptyp_var _ | Ptyp_object _ | Ptyp_class _ |Ptyp_variant _ | Ptyp_poly _ | Ptyp_package _ | Ptyp_extension _ -> - None ) + None + | Ptyp_constr_unboxed (_, _ :: _ :: _) -> Some (Comma, Non) + | Ptyp_constr_unboxed _ -> Some (Apply, Non) ) | {ctx= Cty {pcty_desc; _}; ast= Typ typ; _} -> ( match pcty_desc with | Pcty_constr (_, _ :: _ :: _) -> Some (Comma, Non) @@ -1597,64 +1705,67 @@ end = struct | _ -> None ) | {ast= Cty _; _} -> None | {ast= Typ _; _} -> None - | {ctx= Exp {pexp_desc; _}; ast= Exp exp} -> ( - match pexp_desc with - | Pexp_tuple (e0 :: _) -> - Some (Comma, if exp == e0 then Left else Right) - | Pexp_cons l -> - Some (ColonColon, if exp == List.last_exn l then Right else Left) - | Pexp_construct - ({txt= Lident "[]"; _}, Some {pexp_desc= Pexp_tuple [_; _]; _}) -> - Some (Semi, Non) - | Pexp_array _ | Pexp_list _ -> Some (Semi, Non) - | Pexp_construct (_, Some _) - |Pexp_assert _ | Pexp_lazy _ - |Pexp_variant (_, Some _) -> - Some (Apply, Non) - | Pexp_indexop_access {pia_lhs= lhs; pia_rhs= rhs; _} -> ( - if lhs == exp then Some (Dot, Left) - else - match rhs with - | Some e when e == exp -> Some (LessMinus, Right) - | _ -> Some (Low, Left) ) - | Pexp_prefix ({txt= i; loc}, _) -> ( - match i with - | "~-" | "~-." | "~+" | "~+." -> - if - loc.loc_end.pos_cnum - loc.loc_start.pos_cnum - = String.length i - 1 - then Some (UMinus, Non) - else Some (High, Non) - | _ -> ( - match i.[0] with - | '!' | '?' | '~' -> Some (High, Non) - | _ -> Some (Apply, Non) ) ) - | Pexp_infix ({txt= i; _}, e1, _) -> ( - let child = if e1 == exp then Left else Right in - match (i.[0], i) with - | _, ":=" -> Some (ColonEqual, child) - | _, ("or" | "||") -> Some (BarBar, child) - | _, ("&" | "&&") -> Some (AmperAmper, child) - | ('=' | '<' | '>' | '|' | '&' | '$'), _ | _, "!=" -> - Some (InfixOp0, child) - | ('@' | '^'), _ -> Some (InfixOp1, child) - | ('+' | '-'), _ -> Some (InfixOp2, child) - | '*', _ when String.(i <> "*") && Char.(i.[1] = '*') -> - Some (InfixOp4, child) - | ('*' | '/' | '%'), _ | _, ("lor" | "lxor" | "mod" | "land") -> - Some (InfixOp3, child) - | _, ("lsl" | "lsr" | "asr") -> Some (InfixOp4, child) - | '#', _ -> Some (HashOp, child) - | _ -> Some (Apply, child) ) - | Pexp_apply _ -> Some (Apply, Non) - | Pexp_setfield (e0, _, _) when e0 == exp -> Some (Dot, Left) - | Pexp_setfield (_, _, e0) when e0 == exp -> Some (LessMinus, Non) - | Pexp_setinstvar _ -> Some (LessMinus, Non) - | Pexp_field _ -> Some (Dot, Left) - (* We use [Dot] so [x#y] has the same precedence as [x.y], it is - different to what is done in the parser, but it is intended. *) - | Pexp_send _ -> Some (Dot, Left) - | _ -> None ) + | {ctx= Exp ({pexp_desc; _} as ctx); ast= Exp exp} -> ( + Exp.maybe_extension ctx prec_ctx_extension + @@ fun () -> + match pexp_desc with + | Pexp_tuple (e0 :: _) -> + Some (Comma, if exp == e0 then Left else Right) + | Pexp_cons l -> + Some (ColonColon, if exp == List.last_exn l then Right else Left) + | Pexp_construct + ({txt= Lident "[]"; _}, Some {pexp_desc= Pexp_tuple [_; _]; _}) + -> + Some (Semi, Non) + | Pexp_array _ | Pexp_list _ -> Some (Semi, Non) + | Pexp_construct (_, Some _) + |Pexp_assert _ | Pexp_lazy _ + |Pexp_variant (_, Some _) -> + Some (Apply, Non) + | Pexp_indexop_access {pia_lhs= lhs; pia_rhs= rhs; _} -> ( + if lhs == exp then Some (Dot, Left) + else + match rhs with + | Some e when e == exp -> Some (LessMinus, Right) + | _ -> Some (Low, Left) ) + | Pexp_prefix ({txt= i; loc}, _) -> ( + match i with + | "~-" | "~-." | "~+" | "~+." -> + if + loc.loc_end.pos_cnum - loc.loc_start.pos_cnum + = String.length i - 1 + then Some (UMinus, Non) + else Some (High, Non) + | _ -> ( + match i.[0] with + | '!' | '?' | '~' -> Some (High, Non) + | _ -> Some (Apply, Non) ) ) + | Pexp_infix ({txt= i; _}, e1, _) -> ( + let child = if e1 == exp then Left else Right in + match (i.[0], i) with + | _, ":=" -> Some (ColonEqual, child) + | _, ("or" | "||") -> Some (BarBar, child) + | _, ("&" | "&&") -> Some (AmperAmper, child) + | ('=' | '<' | '>' | '|' | '&' | '$'), _ | _, "!=" -> + Some (InfixOp0, child) + | ('@' | '^'), _ -> Some (InfixOp1, child) + | ('+' | '-'), _ -> Some (InfixOp2, child) + | '*', _ when String.(i <> "*") && Char.(i.[1] = '*') -> + Some (InfixOp4, child) + | ('*' | '/' | '%'), _ | _, ("lor" | "lxor" | "mod" | "land") -> + Some (InfixOp3, child) + | _, ("lsl" | "lsr" | "asr") -> Some (InfixOp4, child) + | '#', _ -> Some (HashOp, child) + | _ -> Some (Apply, child) ) + | Pexp_apply _ -> Some (Apply, Non) + | Pexp_setfield (e0, _, _) when e0 == exp -> Some (Dot, Left) + | Pexp_setfield (_, _, e0) when e0 == exp -> Some (LessMinus, Non) + | Pexp_setinstvar _ -> Some (LessMinus, Non) + | Pexp_field _ -> Some (Dot, Left) + (* We use [Dot] so [x#y] has the same precedence as [x.y], it is + different to what is done in the parser, but it is intended. *) + | Pexp_send _ -> Some (Dot, Left) + | _ -> None ) | {ctx= Cl {pcl_desc; _}; ast= Cl _ | Exp _} -> ( match pcl_desc with Pcl_apply _ -> Some (Apply, Non) | _ -> None ) | { ctx= Exp _ @@ -1691,7 +1802,8 @@ end = struct | Ptyp_alias _ -> Some As | Ptyp_any | Ptyp_var _ | Ptyp_constr _ | Ptyp_object _ |Ptyp_class _ | Ptyp_variant _ | Ptyp_poly _ | Ptyp_extension _ -> - None ) + None + | Ptyp_constr_unboxed _ -> None ) | Td _ -> None | Cty {pcty_desc; _} -> ( match pcty_desc with Pcty_arrow _ -> Some MinusGreater | _ -> None ) @@ -1804,6 +1916,13 @@ end = struct ( Str {pstr_desc= Pstr_exception _; _} | Sig {psig_desc= Psig_exception _; _} ) } -> true + | {ast= {ptyp_desc= Ptyp_arrow _; ptyp_attributes= attrs; _}; _} + when List.exists attrs ~f:(fun a -> + String.equal a.attr_name.txt "extension.curry" ) -> + true + | { ast= {ptyp_desc= Ptyp_poly _; _} + ; ctx= Typ {ptyp_desc= Ptyp_arrow _; _} } -> + true | _ -> ( match ambig_prec (sub_ast ~ctx (Typ typ)) with | `Ambiguous -> true @@ -1824,6 +1943,7 @@ end = struct || match (ctx, mty.pmty_desc) with | Mty {pmty_desc= Pmty_with _; _}, Pmty_with _ -> true + | Mty {pmty_desc= Pmty_strengthen _; _}, Pmty_strengthen _ -> true | _ -> false (** [parenze_mod {ctx; ast}] holds when module expr [ast] should be @@ -1847,12 +1967,17 @@ end = struct | Ppat_tuple _ -> true | _ -> false + let parenze_pat_extension (epat : Extensions.Pattern.t) = + match epat with Epat_immutable_array (Iapat_immutable_array _) -> false + (** [parenze_pat {ctx; ast}] holds when pattern [ast] should be parenthesized in context [ctx]. *) let parenze_pat ({ctx; ast= pat} as xpat) = assert_check_pat xpat ; - Pat.has_trailing_attributes pat - || + (fun k -> Pat.has_trailing_attributes pat || k ()) + @@ fun () -> + Pat.maybe_extension pat parenze_pat_extension + @@ fun () -> match (ctx, pat.ppat_desc) with | Pat {ppat_desc= Ppat_cons pl; _}, Ppat_cons _ when List.last_exn pl == pat -> @@ -1863,6 +1988,9 @@ end = struct | Ppat_construct _ | Ppat_record _ | Ppat_variant _ -> false | _ -> true ) | Pat {ppat_desc= Ppat_construct _; _}, Ppat_cons _ -> true + | ( Exp {pexp_desc= Pexp_fun _; _} + , Ppat_constraint (_, {ptyp_desc= Ptyp_poly _; _}) ) -> + true | _, Ppat_constraint (_, {ptyp_desc= Ptyp_poly _; _}) -> false | ( Exp {pexp_desc= Pexp_letop _; _} , ( Ppat_construct (_, Some _) @@ -1871,6 +1999,8 @@ end = struct | Ppat_or _ | Ppat_alias _ | Ppat_constraint ({ppat_desc= Ppat_any; _}, _) ) ) -> true + | Lb _, Ppat_constraint ({ppat_desc= Ppat_any; _}, _) -> true + | Lb _, Ppat_constraint ({ppat_desc= Ppat_tuple _; _}, _) -> false | ( Exp {pexp_desc= Pexp_letop _; _} , Ppat_constraint ({ppat_desc= Ppat_tuple _; _}, _) ) -> false @@ -1950,12 +2080,22 @@ end = struct (* exponential without memoization *) let memo = Hashtbl.Poly.create () in register_reset (fun () -> Hashtbl.clear memo) ; - fun cls exp -> + fun c cls exp -> let exposed_ () = let continue subexp = - (not (parenze_exp (sub_exp ~ctx:(Exp exp) subexp))) - && exposed_right_exp cls subexp + (not (parenze_exp c (sub_exp c ~ctx:(Exp exp) subexp))) + && exposed_right_exp c cls subexp + in + let exposed_extension eexp = + (* Inlined because we're closed over [memo] *) + match eexp with + | Extensions.Expression.Eexp_comprehension + (Cexp_list_comprehension _ | Cexp_array_comprehension _) + |Eexp_immutable_array (Iaexp_immutable_array _) -> + false in + Exp.maybe_extension exp exposed_extension + @@ fun () -> match exp.pexp_desc with | Pexp_assert e |Pexp_construct (_, Some e) @@ -1991,6 +2131,13 @@ end = struct | Pexp_function cases | Pexp_match (_, cases) | Pexp_try (_, cases) -> continue (List.last_exn cases).pc_rhs + | Pexp_apply + ( { pexp_desc= + Pexp_extension ({txt= "extension.local"; _}, PStr []) + ; _ } + , [(Nolabel, _)] ) + when match cls with Then -> true | _ -> false -> + true | Pexp_apply (_, args) -> continue (snd (List.last_exn args)) | Pexp_tuple es -> continue (List.last_exn es) | Pexp_array _ | Pexp_list _ | Pexp_coerce _ | Pexp_constant _ @@ -2010,28 +2157,38 @@ end = struct and exposed_right_cl = let memo = Hashtbl.Poly.create () in register_reset (fun () -> Hashtbl.clear memo) ; - fun cls cl -> + fun c cls cl -> let exposed_ () = match cl.pcl_desc with | Pcl_apply (_, args) -> let exp = snd (List.last_exn args) in - (not (parenze_exp (sub_exp ~ctx:(Cl cl) exp))) - && exposed_right_exp cls exp + (not (parenze_exp c (sub_exp c ~ctx:(Cl cl) exp))) + && exposed_right_exp c cls exp | Pcl_fun (_, _, _, e) -> - (not (parenze_cl (sub_cl ~ctx:(Cl cl) e))) - && exposed_right_cl cls e + (not (parenze_cl c (sub_cl ~ctx:(Cl cl) e))) + && exposed_right_cl c cls e | _ -> false in Cl.mem_cls cls cl || Hashtbl.find_or_add memo (cls, cl) ~default:exposed_ - and mark_parenzed_inner_nested_match exp = + and mark_parenzed_inner_nested_match c exp = let exposed_ () = let continue subexp = - if not (parenze_exp (sub_exp ~ctx:(Exp exp) subexp)) then - mark_parenzed_inner_nested_match subexp ; + if not (parenze_exp c (sub_exp c ~ctx:(Exp exp) subexp)) then + mark_parenzed_inner_nested_match c subexp ; false in + let exposed_extension eexp = + (* Inlined because we're nested *) + match eexp with + | Extensions.Expression.Eexp_comprehension + (Cexp_list_comprehension _ | Cexp_array_comprehension _) + |Eexp_immutable_array (Iaexp_immutable_array _) -> + false + in + Exp.maybe_extension exp exposed_extension + @@ fun () -> match exp.pexp_desc with | Pexp_assert e |Pexp_construct (_, Some e) @@ -2061,12 +2218,12 @@ end = struct | Pexp_function cases | Pexp_match (_, cases) | Pexp_try (_, cases) -> List.iter cases ~f:(fun case -> - mark_parenzed_inner_nested_match case.pc_rhs ) ; + mark_parenzed_inner_nested_match c case.pc_rhs ) ; true | _ -> continue e ) | Pexp_function cases | Pexp_match (_, cases) | Pexp_try (_, cases) -> List.iter cases ~f:(fun case -> - mark_parenzed_inner_nested_match case.pc_rhs ) ; + mark_parenzed_inner_nested_match c case.pc_rhs ) ; true | Pexp_indexop_access {pia_rhs= rhs; _} -> ( match rhs with Some e -> continue e | None -> false ) @@ -2088,7 +2245,7 @@ end = struct (** [parenze_exp {ctx; ast}] holds when expression [ast] should be parenthesized in context [ctx]. *) - and parenze_exp ({ctx; ast= exp} as xexp) = + and parenze_exp c ({ctx; ast= exp} as xexp) = let parenze () = let is_right_infix_arg ctx_desc exp = match ctx_desc with @@ -2107,8 +2264,8 @@ end = struct match ctx with | Exp {pexp_desc; _} -> if is_right_infix_arg pexp_desc exp then Exp.is_sequence exp - else exposed_right_exp Non_apply exp - | _ -> exposed_right_exp Non_apply exp ) + else exposed_right_exp c Non_apply exp + | _ -> exposed_right_exp c Non_apply exp ) in let rec ifthenelse pexp_desc = match pexp_desc with @@ -2130,16 +2287,80 @@ end = struct , _ ) when lhs == exp -> true - | _ when lhs == exp -> exposed_right_exp Let_match exp + | _ when lhs == exp -> exposed_right_exp c Let_match exp | _ when rhs == exp -> false | _ -> failwith "exp must be lhs or rhs from the parent expression" in + let trailing_attrs_require_parens ctx exp = + match ctx with + | Exp {pexp_desc; _} -> ( + match pexp_desc with + | Pexp_let (_, e) + |Pexp_letmodule (_, _, _, e) + |Pexp_letexception (_, e) + |Pexp_letopen (_, e) + |Pexp_open (_, e) + |Pexp_fun (_, _, _, e) + |Pexp_newtype (_, e) + |Pexp_constraint (e, _) + |Pexp_coerce (e, _, _) + when e == exp -> + false + | Pexp_let (pvbs, _) + when List.exists pvbs.pvbs_bindings ~f:(fun pvb -> + pvb.pvb_expr == exp ) -> + false + | _ -> true ) + | _ -> true + in assert_check_exp xexp ; Hashtbl.find marked_parenzed_inner_nested_match exp |> Option.value ~default:false || + (* We don't just do the full-scale match on [Extensions.Expression.of_ast + exp] here because the following match is on [ctx, exp] and is very + complicated. These booleans let us integrate our extended AST nodes + into the match in the most natural possible way relative to the + structure of the existing match. We make them lazy because we only + want to run [of_ast] if we know that [exp] isn't actually a subterm of + an extension expression. *) + let opt_eexp = lazy (Extensions.Expression.of_ast exp) in + let is_extension_comprehension = + Lazy.map opt_eexp ~f:(function + | Some (Eexp_comprehension _) -> true + | _ -> false ) + in + let is_extension_immutable_array = + Lazy.map opt_eexp ~f:(function + | Some (Eexp_immutable_array _) -> true + | _ -> false ) + in match (ctx, exp) with | Str {pstr_desc= Pstr_eval _; _}, _ -> false + | ( Exp + { pexp_desc= + Pexp_apply + ( { pexp_desc= + Pexp_extension ({txt= extension_local; _}, PStr []) + ; _ } + , [(Nolabel, _)] ) + ; _ } + , _ ) + when Conf.is_jane_street_local_annotation c "local" + ~test:extension_local -> + false + | ( Exp + { pexp_desc= + Pexp_apply + ( { pexp_desc= + Pexp_extension ({txt= extension_exclave; _}, PStr []) + ; _ } + , [(Nolabel, _)] ) + ; _ } + , _ ) + when Conf.is_jane_street_local_annotation c "exclave" + ~test:extension_exclave -> + false | _, {pexp_desc= Pexp_infix _; pexp_attributes= _ :: _; _} -> true | ( Str { pstr_desc= @@ -2226,96 +2447,143 @@ end = struct , {pexp_desc= Pexp_construct _ | Pexp_cons _; _} ) when e == exp -> true - | Exp {pexp_desc; _}, _ -> ( - match pexp_desc with - | Pexp_extension - ( _ - , PStr - [ { pstr_desc= - Pstr_eval - ( { pexp_desc= - ( Pexp_function cases - | Pexp_match (_, cases) - | Pexp_try (_, cases) ) - ; _ } - , _ ) - ; _ } ] ) - |Pexp_function cases - |Pexp_match (_, cases) - |Pexp_try (_, cases) -> - if !leading_nested_match_parens then - List.iter cases ~f:(fun {pc_rhs; _} -> - mark_parenzed_inner_nested_match pc_rhs ) ; - List.exists cases ~f:(fun {pc_rhs; _} -> pc_rhs == exp) - && exposed_right_exp Match exp - | Pexp_ifthenelse (eN, _) - when List.exists eN ~f:(fun x -> x.if_cond == exp) -> - false - | Pexp_ifthenelse (eN, None) when (List.last_exn eN).if_body == exp -> - exposed_right_exp Then exp - | Pexp_ifthenelse (eN, _) - when List.exists eN ~f:(fun x -> x.if_body == exp) -> - exposed_right_exp ThenElse exp - | Pexp_ifthenelse (_, Some els) when els == exp -> Exp.is_sequence exp - | Pexp_apply (({pexp_desc= Pexp_new _; _} as exp2), _) when exp2 == exp - -> - false - | Pexp_apply - ( ( { pexp_desc= - Pexp_extension - ( _ - , PStr - [ { pstr_desc= - Pstr_eval ({pexp_desc= Pexp_new _; _}, []) - ; _ } ] ) - ; _ } as exp2 ) - , _ ) - when exp2 == exp -> - false - | Pexp_record (flds, _) - when List.exists flds ~f:(fun (_, _, e0) -> - Option.exists e0 ~f:(fun x -> x == exp) ) -> - exposed_right_exp Non_apply exp - (* Non_apply is perhaps pessimistic *) - | Pexp_record (_, Some ({pexp_desc= Pexp_prefix _; _} as e0)) - when e0 == exp -> - (* don't put parens around [!e] in [{ !e with a; b }] *) - false - | Pexp_record - ( _ - , Some - ( { pexp_desc= - ( Pexp_ident _ | Pexp_constant _ | Pexp_record _ - | Pexp_field _ ) - ; _ } as e0 ) ) - when e0 == exp -> - false - | Pexp_record (_, Some e0) when e0 == exp -> true - | Pexp_sequence (lhs, rhs) -> exp_in_sequence lhs rhs exp - | Pexp_apply (_, args) - when List.exists args ~f:(fun (_, e0) -> - match (e0.pexp_desc, e0.pexp_attributes) with - | Pexp_list _, _ :: _ when e0 == exp -> true - | Pexp_array _, _ :: _ when e0 == exp -> true - | _ -> false ) -> - true - | _ -> ( - match exp.pexp_desc with - | Pexp_list _ | Pexp_array _ -> false - | _ -> Exp.has_trailing_attributes exp || parenze () ) ) + | Exp ({pexp_desc; _} as ctx_exp), _ -> ( + (* This is the old fallthrough case, but we need it for the + extensions match and the real match, so we factor it out; we have + to do an external match on [ctx_exp] because the below fallthrough + case matches on subterms, which can cause exceptions if we break + into the representation of a language extension. *) + let fallthrough_case () = + match exp.pexp_desc with + | Pexp_list _ | Pexp_array _ -> false + | _ + when Lazy.force is_extension_comprehension + || Lazy.force is_extension_immutable_array -> + false + | _ -> + Exp.has_trailing_attributes exp + && trailing_attrs_require_parens ctx exp + || parenze () + in + let parenze_extension ctx_eexp = + match ctx_eexp with + | Extensions.Expression.Eexp_comprehension _ + |Eexp_immutable_array _ -> + fallthrough_case () + in + Exp.maybe_extension ctx_exp parenze_extension + @@ fun () -> + match pexp_desc with + | Pexp_extension + ( _ + , PStr + [ { pstr_desc= + Pstr_eval + ( { pexp_desc= + ( Pexp_function cases + | Pexp_match (_, cases) + | Pexp_try (_, cases) ) + ; _ } + , _ ) + ; _ } ] ) + |Pexp_function cases + |Pexp_match (_, cases) + |Pexp_try (_, cases) -> + if !leading_nested_match_parens then + List.iter cases ~f:(fun {pc_rhs; _} -> + mark_parenzed_inner_nested_match c pc_rhs ) ; + List.exists cases ~f:(fun {pc_rhs; _} -> pc_rhs == exp) + && exposed_right_exp c Match exp + | Pexp_ifthenelse (eN, _) + when List.exists eN ~f:(fun x -> x.if_cond == exp) -> + false + | Pexp_ifthenelse (eN, None) when (List.last_exn eN).if_body == exp + -> + exposed_right_exp c Then exp + | Pexp_ifthenelse (eN, _) + when List.exists eN ~f:(fun x -> x.if_body == exp) -> + exposed_right_exp c ThenElse exp + | Pexp_ifthenelse (_, Some els) when els == exp -> + Exp.is_sequence exp + | Pexp_apply (({pexp_desc= Pexp_new _; _} as exp2), _) + when exp2 == exp -> + false + | Pexp_apply + ( ( { pexp_desc= + Pexp_extension + ( _ + , PStr + [ { pstr_desc= + Pstr_eval ({pexp_desc= Pexp_new _; _}, []) + ; _ } ] ) + ; _ } as exp2 ) + , _ ) + when exp2 == exp -> + false + | Pexp_record (flds, _) + when List.exists flds ~f:(fun (_, _, e0) -> + Option.exists e0 ~f:(fun x -> x == exp) ) -> + exposed_right_exp c Non_apply exp + (* Non_apply is perhaps pessimistic *) + | Pexp_record (_, Some ({pexp_desc= Pexp_prefix _; _} as e0)) + when e0 == exp -> + (* don't put parens around [!e] in [{ !e with a; b }] *) + false + | Pexp_record + ( _ + , Some + ( { pexp_desc= + ( Pexp_ident _ | Pexp_constant _ | Pexp_record _ + | Pexp_field _ ) + ; _ } as e0 ) ) + when e0 == exp -> + false + | Pexp_record (_, Some e0) when e0 == exp -> true + | Pexp_sequence (lhs, rhs) -> exp_in_sequence lhs rhs exp + | Pexp_apply (_, args) + when List.exists args ~f:(fun (_, e0) -> + let extension ee0 = + match (ee0, e0.pexp_attributes) with + | ( ( Extensions.Expression.Eexp_comprehension + ( Cexp_list_comprehension _ + | Cexp_array_comprehension _ ) + | Eexp_immutable_array (Iaexp_immutable_array _) ) + , _ :: _ ) + when e0 == exp -> + (* Has to be [e0] and not [ee0], as [ee0] isn't a + true OCaml expression and was just synthesized + afresh *) + true + | _ -> false + in + Exp.maybe_extension e0 extension + @@ fun () -> + match (e0.pexp_desc, e0.pexp_attributes) with + | Pexp_list _, _ :: _ when e0 == exp -> true + | Pexp_array _, _ :: _ when e0 == exp -> true + | _ -> false ) -> + true + | _ -> fallthrough_case () ) | _, {pexp_desc= Pexp_list _; _} -> false | _, {pexp_desc= Pexp_array _; _} -> false - | _, exp when Exp.has_trailing_attributes exp -> true + | _, _ + when Lazy.force is_extension_comprehension + || Lazy.force is_extension_immutable_array -> + false + | ctx, exp + when Exp.has_trailing_attributes exp + && trailing_attrs_require_parens ctx exp -> + true | _ -> false (** [parenze_cl {ctx; ast}] holds when class expr [ast] should be parenthesized in context [ctx]. *) - and parenze_cl ({ctx; ast= cl} as xcl) = + and parenze_cl c ({ctx; ast= cl} as xcl) = assert_check_cl xcl ; match ambig_prec (sub_ast ~ctx (Cl cl)) with | `No_prec_ctx -> false | `Ambiguous -> true - | _ -> exposed_right_cl Non_apply cl + | _ -> exposed_right_cl c Non_apply cl let parenze_nested_exp {ctx; ast= exp} = let infix_prec ast = diff --git a/lib/Ast.mli b/lib/Ast.mli index 2c81d469ae..0d7dc4a35d 100644 --- a/lib/Ast.mli +++ b/lib/Ast.mli @@ -152,7 +152,7 @@ val sub_cty : ctx:t -> class_type -> class_type xt val sub_pat : ctx:t -> pattern -> pattern xt (** Construct a pattern-in-context. *) -val sub_exp : ctx:t -> expression -> expression xt +val sub_exp : Conf.t -> ctx:t -> expression -> expression xt (** Construct a expression-in-context. *) val sub_cl : ctx:t -> class_expr -> class_expr xt @@ -189,7 +189,7 @@ val is_simple : Conf.t -> (expression xt -> int) -> expression xt -> bool (** 'Classes' of expressions which are parenthesized differently. *) type cls = Let_match | Match | Non_apply | Sequence | Then | ThenElse -val exposed_right_exp : cls -> expression -> bool +val exposed_right_exp : Conf.t -> cls -> expression -> bool (** [exposed_right_exp cls exp] holds if there is a right-most subexpression of [exp] which is of class [cls] and is not parenthesized. *) @@ -205,7 +205,7 @@ val parenze_cty : class_type xt -> bool (** [parenze_cty xcty] holds when class_type-in-context [xcty] should be parenthesized. *) -val parenze_cl : class_expr xt -> bool +val parenze_cl : Conf.t -> class_expr xt -> bool (** [parenze_cl xcl] holds when class-in-context [xcl] should be parenthesized. *) @@ -213,7 +213,7 @@ val parenze_pat : pattern xt -> bool (** [parenze_pat xpat] holds when pattern-in-context [xpat] should be parenthesized. *) -val parenze_exp : expression xt -> bool +val parenze_exp : Conf.t -> expression xt -> bool (** [parenze_exp xexp] holds when expression-in-context [xexp] should be parenthesized. *) diff --git a/lib/Conf.ml b/lib/Conf.ml index be35cd1e53..f1ff894729 100644 --- a/lib/Conf.ml +++ b/lib/Conf.ml @@ -260,7 +260,8 @@ let default = ; ocaml_version= elt Ocaml_version.Releases.v4_04_0 ; quiet= elt false ; disable_conf_attrs= elt false - ; version_check= elt true } } + ; version_check= elt true + ; rewrite_old_style_jane_street_local_annotations= elt false } } module V = struct let v0_12 = Version.make ~major:0 ~minor:12 ~patch:None @@ -1453,6 +1454,23 @@ module Operational = struct (fun conf elt -> update conf ~f:(fun f -> {f with version_check= elt})) (fun conf -> conf.opr_opts.version_check) + let rewrite_old_style_jane_street_local_annotations = + let doc = + "Rewrite all Jane Street annotations for use with the local mode, \ + such as \"[%local]\" or \"[@ocaml.global]\", into their \ + pretty-printed syntactic form, such as \"local_\" or \"global_\". \ + THIS OPTION WILL CHANGE THE RESULTING AST." + in + Decl.flag ~default + ~names:["rewrite-old-style-jane-street-local-annotations"] + ~doc ~kind + (fun conf elt -> + update conf ~f:(fun f -> + {f with rewrite_old_style_jane_street_local_annotations= elt} ) + ) + (fun conf -> + conf.opr_opts.rewrite_old_style_jane_street_local_annotations ) + let options : Store.t = Store. [ elt comment_check @@ -1463,7 +1481,8 @@ module Operational = struct ; elt ocaml_version ; elt quiet ; elt disable_conf_attrs - ; elt version_check ] + ; elt version_check + ; elt rewrite_old_style_jane_street_local_annotations ] end let options = Operational.options @ Formatting.options @ options @@ -1571,6 +1590,13 @@ let parse_state_attr attr = | Ok ("disable", _) -> Some `Disable | _ -> None +let is_jane_street_local_annotation config name ~test = + String.equal test ("extension." ^ name) + || + if config.opr_opts.rewrite_old_style_jane_street_local_annotations.v then + String.equal test name || String.equal test ("ocaml." ^ name) + else false + let print_config = Decl.print_config options let term = Decl.Store.to_term options diff --git a/lib/Conf.mli b/lib/Conf.mli index 8b1d2664d2..385b4acd5c 100644 --- a/lib/Conf.mli +++ b/lib/Conf.mli @@ -25,6 +25,8 @@ val update_state : t -> [`Enable | `Disable] -> t val parse_state_attr : Parsetree.attribute -> [`Enable | `Disable] option +val is_jane_street_local_annotation : t -> string -> test:string -> bool + val parse_line : t -> ?version_check:bool diff --git a/lib/Conf_t.ml b/lib/Conf_t.ml index 59f0c6b458..5eb336b4cb 100644 --- a/lib/Conf_t.ml +++ b/lib/Conf_t.ml @@ -129,7 +129,8 @@ type opr_opts = ; ocaml_version: Ocaml_version.t elt ; quiet: bool elt ; disable_conf_attrs: bool elt - ; version_check: bool elt } + ; version_check: bool elt + ; rewrite_old_style_jane_street_local_annotations: bool elt } type t = { fmt_opts: fmt_opts diff --git a/lib/Conf_t.mli b/lib/Conf_t.mli index 853e321d8e..2b1af20f84 100644 --- a/lib/Conf_t.mli +++ b/lib/Conf_t.mli @@ -132,7 +132,8 @@ type opr_opts = (** Version of OCaml syntax of the output. *) ; quiet: bool elt ; disable_conf_attrs: bool elt - ; version_check: bool elt } + ; version_check: bool elt + ; rewrite_old_style_jane_street_local_annotations: bool elt } type t = { fmt_opts: fmt_opts diff --git a/lib/Extended_ast.ml b/lib/Extended_ast.ml index 34445fd11c..5143dacc41 100644 --- a/lib/Extended_ast.ml +++ b/lib/Extended_ast.ml @@ -10,8 +10,13 @@ (**************************************************************************) open Parser_extended +module Extensions = Extensions include Parsetree +(* Enable all language extensions *) +let () = + List.iter ~f:Language_extension.enable Language_extension.max_compatible + let equal_core_type : core_type -> core_type -> bool = Poly.equal type use_file = toplevel_phrase list diff --git a/lib/Extended_ast.mli b/lib/Extended_ast.mli index 6842408cca..4a8627b4be 100644 --- a/lib/Extended_ast.mli +++ b/lib/Extended_ast.mli @@ -10,6 +10,7 @@ (**************************************************************************) open Parser_extended +module Extensions = Extensions include module type of Parsetree diff --git a/lib/Fmt_ast.ml b/lib/Fmt_ast.ml index 330dd29cdd..609ad3a42a 100644 --- a/lib/Fmt_ast.ml +++ b/lib/Fmt_ast.ml @@ -131,7 +131,7 @@ let fmt_expressions c width sub_exp exprs fmt_expr p loc = | `Fit_or_vertical -> fmt_elements_collection c p Exp.location loc fmt_expr exprs | `Wrap -> - let is_simple x = is_simple c.conf width (sub_exp x) in + let is_simple x = is_simple c.conf width (sub_exp c.conf x) in let break x1 x2 = not (is_simple x1 && is_simple x2) in let grps = List.group exprs ~break in let fmt_grp ~first:first_grp ~last:last_grp exprs = @@ -195,6 +195,14 @@ let update_items_config c items update_config = let _, items = List.fold_map items ~init:c ~f:with_config in items +let check_include_functor_attr attrs = + match + List.partition_tf attrs ~f:(fun attr -> + String.equal attr.attr_name.txt "extension.include_functor" ) + with + | [], _ -> (attrs, false) + | _ :: _, rest -> (rest, true) + let box_semisemi c ~parent_ctx b k = let space = Poly.(c.conf.fmt_opts.sequence_style.v = `Separator) in match parent_ctx with @@ -257,6 +265,10 @@ let fmt_constant c ?epi {pconst_desc; pconst_loc= loc} = Cmts.fmt c loc @@ match pconst_desc with + | Pconst_unboxed_integer (sign, lit, suf) + |Pconst_unboxed_float (sign, lit, suf) -> + (match sign with Positive -> noop | Negative -> char '-') + $ char '#' $ str lit $ opt suf char | Pconst_integer (lit, suf) | Pconst_float (lit, suf) -> str lit $ opt suf char | Pconst_char (_, s) -> wrap "'" "'" @@ str s @@ -533,6 +545,28 @@ let fmt_type_var s = $ fmt_if (String.length s > 1 && Char.equal s.[1] '\'') " " $ str s +let split_global_flags_from_attrs conf atrs = + match + List.partition_map atrs ~f:(fun a -> + if + Conf.is_jane_street_local_annotation conf "global" + ~test:a.attr_name.txt + then First `Global + else Second a ) + with + | [`Global], atrs -> (true, atrs) + | _ -> (false, atrs) + +let is_layout attr = + match attr.attr_name.txt with + | "any" -> true + | "value" -> true + | "void" -> true + | "immediate" -> true + | "immediate64" -> true + | "float64" -> true + | _ -> false + let rec fmt_extension_aux c ctx ~key (ext, pld) = match (ext.txt, pld, ctx) with (* Quoted extensions (since ocaml 4.11). *) @@ -665,13 +699,14 @@ and fmt_payload c ctx pld = | PStr mex -> fmt_if (not (List.is_empty mex)) "@ " $ fmt_structure c ctx mex | PSig mty -> - str ":" - $ fmt_if (not (List.is_empty mty)) "@ " + fmt ":@ " + (* A trailing space is necessary because [:]] is the immutable array + closing delimiter*) $ fmt_signature c ctx mty | PTyp typ -> fmt ":@ " $ fmt_core_type c (sub_typ ~ctx typ) | PPat (pat, exp) -> let fmt_when exp = - str " when " $ fmt_expression c (sub_exp ~ctx exp) + str " when " $ fmt_expression c (sub_exp c.conf ~ctx exp) in fmt "?@ " $ fmt_pattern c (sub_pat ~ctx pat) $ opt exp fmt_when @@ -711,15 +746,17 @@ and type_constr_and_body c xbody = Exp Ast_helper.(Exp.fun_ Nolabel None (Pat.any ()) exp) in ( Some (fmt_type_cstr c ~constraint_ctx:`Fun (sub_typ ~ctx:typ_ctx typ)) - , sub_exp ~ctx:exp_ctx exp ) + , sub_exp c.conf ~ctx:exp_ctx exp ) | _ -> (None, xbody) -and fmt_arrow_param c ctx {pap_label= lI; pap_loc= locI; pap_type= tI} = +and fmt_arrow_param c ctx + ({pap_label= lI; pap_loc= locI; pap_type= tI}, localI) = let arg_label lbl = match lbl with - | Nolabel -> None - | Labelled l -> Some (str l.txt $ fmt ":@,") - | Optional l -> Some (str "?" $ str l.txt $ fmt ":@,") + | Nolabel -> if localI then Some (str "local_ ") else None + | Labelled l -> Some (str l.txt $ fmt ":@," $ fmt_if localI "local_ ") + | Optional l -> + Some (str "?" $ str l.txt $ fmt ":@," $ fmt_if localI "local_ ") in let xtI = sub_typ ~ctx tI in let arg = @@ -759,11 +796,28 @@ and fmt_arrow_type c ~ctx ?indent ~parens ~parent_has_parens args fmt_ret_typ [xtyp] should be parenthesized. [constraint_ctx] gives the higher context of the expression, i.e. if the expression is part of a `fun` expression. *) +(* CR layouts: Instead of having a [tydecl_param] argument here, the right + thing would be for [xtyp] to provide enough information to determine + whether we are printing a type parameter in a typedecl. But it doesn't, + and that change would be a much bigger diff and make rebasing on upstream + harder in the future. When layouts are upstreamed and upstream ocamlformat + gets support for them, we should remove tydecl_param and go with whatever + their solution is. *) and fmt_core_type c ?(box = true) ?pro ?(pro_space = true) ?constraint_ctx - ({ast= typ; ctx} as xtyp) = + ?(tydecl_param = false) ({ast= typ; ctx} as xtyp) = protect c (Typ typ) @@ let {ptyp_desc; ptyp_attributes; ptyp_loc; _} = typ in + let ptyp_attributes = + List.filter ptyp_attributes ~f:(fun a -> + not (String.equal a.attr_name.txt "extension.curry") ) + in + let ptyp_attributes = + List.filter ptyp_attributes ~f:(fun a -> + not + (Conf.is_jane_street_local_annotation c.conf "global" + ~test:a.attr_name.txt ) ) + in update_config_maybe_disabled c ptyp_loc ptyp_attributes @@ fun c -> ( match pro with @@ -776,12 +830,20 @@ and fmt_core_type c ?(box = true) ?pro ?(pro_space = true) ?constraint_ctx let doc, atrs = doc_atrs ptyp_attributes in Cmts.fmt c ptyp_loc @@ (fun k -> k $ fmt_docstring c ~pro:(fmt "@ ") doc) - @@ ( if List.is_empty atrs then Fn.id - else fun k -> - hvbox 0 (Params.parens c.conf (k $ fmt_attributes c ~pre:Cut atrs)) - ) + @@ ( match atrs with + | [] -> Fn.id + | [attr] when is_layout attr -> + Fn.id + (* CR layouts v1.5: layout annotations on type params are printed by + the type parameter printer. Revisit when we have support for + pretty layout annotations in more places. *) + | _ -> + fun k -> + hvbox 0 + (Params.parens_if (not tydecl_param) c.conf + (k $ fmt_attributes c ~pre:Cut atrs) ) ) @@ - let parens = parenze_typ xtyp in + let parens = (not tydecl_param) && parenze_typ xtyp in hvbox_if box 0 @@ Params.parens_if (match typ.ptyp_desc with Ptyp_tuple _ -> false | _ -> parens) @@ -809,6 +871,9 @@ and fmt_core_type c ?(box = true) ?pro ?(pro_space = true) ?constraint_ctx | Ptyp_arrow (args, ret_typ) -> Cmts.relocate c.cmts ~src:ptyp_loc ~before:(List.hd_exn args).pap_type.ptyp_loc ~after:ret_typ.ptyp_loc ; + let args, ret_typ, ctx = + Sugar.decompose_arrow c.conf ctx args ret_typ + in let indent = match pro with | Some pro when c.conf.fmt_opts.ocp_indent_compat.v -> @@ -821,7 +886,7 @@ and fmt_core_type c ?(box = true) ?pro ?(pro_space = true) ?constraint_ctx (String.make (Int.max 1 (indent - String.length pro)) ' ') ) | _ -> None in - let fmt_ret_typ = fmt_core_type c (sub_typ ~ctx ret_typ) in + let fmt_ret_typ = fmt_arrow_param c ctx ret_typ in fmt_arrow_type c ~ctx ?indent ~parens:parenze_constraint_ctx ~parent_has_parens:parens args (Some fmt_ret_typ) | Ptyp_constr (lid, []) -> fmt_longident_loc c lid @@ -941,6 +1006,15 @@ and fmt_core_type c ?(box = true) ?pro ?(pro_space = true) ?constraint_ctx (sub_typ ~ctx >> fmt_core_type c) ) $ fmt "@ " $ fmt_longident_loc c ~pre:"#" lid + | Ptyp_constr_unboxed (lid, []) -> fmt_longident_loc c lid $ char '#' + | Ptyp_constr_unboxed (lid, [t1]) -> + fmt_core_type c (sub_typ ~ctx t1) + $ fmt "@ " $ fmt_longident_loc c lid $ char '#' + | Ptyp_constr_unboxed (lid, t1N) -> + wrap_fits_breaks c.conf "(" ")" + (list t1N (Params.comma_sep c.conf) + (sub_typ ~ctx >> fmt_core_type c) ) + $ fmt "@ " $ fmt_longident_loc c lid $ char '#' and fmt_package_type c ctx cnstrs = let fmt_cstr ~first ~last:_ (lid, typ) = @@ -989,6 +1063,14 @@ and fmt_pattern_attributes c xpat k = Params.parens_if parens_attr c.conf (k $ fmt_attributes c ~pre:Space attrs) +and maybe_fmt_pattern_extension ~ext c ~pro ~parens ~box ~ctx0 ~ctx ~ppat_loc + pat fmt_normal_pattern = + match Extensions.Pattern.of_ast pat with + | Some epat -> + fmt_pattern_extension ~ext c ~pro ~parens ~box ~ctx0 ~ctx ~ppat_loc + epat + | None -> fmt_normal_pattern () + and fmt_pattern ?ext c ?pro ?parens ?(box = false) ({ctx= ctx0; ast= pat} as xpat) = protect c (Pat pat) @@ -1004,7 +1086,9 @@ and fmt_pattern ?ext c ?pro ?parens ?(box = false) | _ -> fun k -> Cmts.fmt c ppat_loc @@ (fmt_opt pro $ k) ) @@ hovbox_if box 0 @@ fmt_pattern_attributes c xpat - @@ + @@ maybe_fmt_pattern_extension ~ext c ~pro ~parens ~box ~ctx0 ~ctx + ~ppat_loc pat + @@ fun () -> match ppat_desc with | Ppat_any -> str "_" | Ppat_var {txt; loc} -> @@ -1219,13 +1303,19 @@ and fmt_pattern ?ext c ?pro ?parens ?(box = false) | Ppat_extension ext -> hvbox c.conf.fmt_opts.extension_indent.v (fmt_extension c ctx ext) | Ppat_open (lid, pat) -> + let can_skip_parens_extension : Extensions.Pattern.t -> _ = function + | Epat_immutable_array (Iapat_immutable_array _) -> true + in let can_skip_parens = - match pat.ppat_desc with - | Ppat_array _ | Ppat_list _ | Ppat_record _ -> true - | Ppat_tuple _ -> - Poly.(c.conf.fmt_opts.parens_tuple_patterns.v = `Always) - | Ppat_construct ({txt= Lident "[]"; _}, None) -> true - | _ -> false + match Extensions.Pattern.of_ast pat with + | Some epat -> can_skip_parens_extension epat + | None -> ( + match pat.ppat_desc with + | Ppat_array _ | Ppat_list _ | Ppat_record _ -> true + | Ppat_tuple _ -> + Poly.(c.conf.fmt_opts.parens_tuple_patterns.v = `Always) + | Ppat_construct ({txt= Lident "[]"; _}, None) -> true + | _ -> false ) in let opn, cls = if can_skip_parens then (".", "") else (".(", ")") in cbox 0 @@ -1233,11 +1323,24 @@ and fmt_pattern ?ext c ?pro ?parens ?(box = false) $ wrap_k (str opn) (str cls) (fmt "@;<0 2>" $ fmt_pattern c (sub_pat ~ctx pat)) ) +and fmt_pattern_extension ~ext:_ c ~pro:_ ~parens:_ ~box:_ ~ctx0 ~ctx + ~ppat_loc : Extensions.Pattern.t -> _ = function + | Epat_immutable_array (Iapat_immutable_array []) -> + hvbox 0 + (wrap_fits_breaks c.conf "[:" ":]" (Cmts.fmt_within c ppat_loc)) + | Epat_immutable_array (Iapat_immutable_array pats) -> + let p = Params.get_iarray_pat c.conf ~ctx:ctx0 in + p.box + (fmt_elements_collection c p Pat.location ppat_loc + (sub_pat ~ctx >> fmt_pattern c >> hvbox 0) + pats ) + and fmt_fun_args c args = let fmt_fun_arg (a : Sugar.arg_kind) = match a with | Val - ( ((Labelled l | Optional l) as lbl) + ( islocal + , ((Labelled l | Optional l) as lbl) , ( { ast= { ppat_desc= ( Ppat_var {txt; loc= _} @@ -1252,37 +1355,60 @@ and fmt_fun_args c args = , None ) when String.equal l.txt txt -> let symbol = match lbl with Labelled _ -> "~" | _ -> "?" in - cbox 0 (str symbol $ fmt_pattern ~box:true c xpat) - | Val ((Optional _ as lbl), xpat, None) -> + cbox 0 + ( str symbol + $ hovbox 0 + (Params.parens_if + (islocal || parenze_pat xpat) + c.conf + (fmt_if islocal "local_ " $ fmt_pattern ~parens:false c xpat) ) + ) + | Val (islocal, (Optional _ as lbl), xpat, None) -> let has_attr = not (List.is_empty xpat.ast.ppat_attributes) in let outer_parens, inner_parens = - match xpat.ast.ppat_desc with - | Ppat_any | Ppat_var _ -> (false, false) - | Ppat_unpack _ -> (not has_attr, true) - | Ppat_tuple _ -> (false, true) - | Ppat_or _ -> (has_attr, true) - | _ -> (not has_attr, false) + match Extensions.Pattern.of_ast xpat.ast with + | Some epat -> ( + (* Inlined because there's nowhere better *) + match epat with + (* Same as the fallthrough case below *) + | Epat_immutable_array (Iapat_immutable_array _) -> + (not has_attr, false) ) + | None -> ( + match xpat.ast.ppat_desc with + | Ppat_any | Ppat_var _ -> (false, false) + | Ppat_unpack _ -> (not has_attr, true) + | Ppat_tuple _ -> (false, true) + | Ppat_or _ -> (has_attr, true) + | _ -> (not has_attr, false) ) in + let outer_parens = outer_parens || islocal in cbox 2 ( fmt_label lbl ":@," $ hovbox 0 @@ Params.parens_if outer_parens c.conf - (fmt_pattern ~parens:inner_parens c xpat) ) - | Val (((Labelled _ | Nolabel) as lbl), xpat, None) -> - cbox 2 (fmt_label lbl ":@," $ fmt_pattern c xpat) + ( fmt_if islocal "local_ " + $ fmt_pattern ~parens:inner_parens c xpat ) ) + | Val (islocal, ((Labelled _ | Nolabel) as lbl), xpat, None) -> + cbox 2 + ( fmt_label lbl ":@," + $ Params.parens_if islocal c.conf + (fmt_if islocal "local_ " $ fmt_pattern c xpat) ) | Val - ( Optional l + ( islocal + , Optional l , ( { ast= {ppat_desc= Ppat_var {txt; loc= _}; ppat_attributes= []; _} ; _ } as xpat ) , Some xexp ) when String.equal l.txt txt -> cbox 0 (wrap "?(" ")" - ( fmt_pattern c ~box:true xpat + ( fmt_if islocal "local_ " + $ fmt_pattern c ~box:true xpat $ fmt " =@;<1 2>" $ hovbox 2 (fmt_expression c xexp) ) ) | Val - ( Optional l + ( islocal + , Optional l , ( { ast= { ppat_desc= Ppat_constraint @@ -1294,9 +1420,10 @@ and fmt_fun_args c args = when String.equal l.txt txt -> cbox 0 (wrap "?(" ")" - ( fmt_pattern c ~parens:false ~box:true xpat + ( fmt_if islocal "local_ " + $ fmt_pattern c ~parens:false ~box:true xpat $ fmt " =@;<1 2>" $ fmt_expression c xexp ) ) - | Val (Optional l, xpat, Some xexp) -> + | Val (islocal, Optional l, xpat, Some xexp) -> let parens = match xpat.ast.ppat_desc with | Ppat_unpack _ -> None @@ -1305,9 +1432,10 @@ and fmt_fun_args c args = cbox 2 ( str "?" $ str l.txt $ wrap_k (fmt ":@,(") (str ")") - ( fmt_pattern c ?parens ~box:true xpat + ( fmt_if islocal "local_ " + $ fmt_pattern c ?parens ~box:true xpat $ fmt " =@;<1 2>" $ fmt_expression c xexp ) ) - | Val ((Labelled _ | Nolabel), _, Some _) -> + | Val (_, (Labelled _ | Nolabel), _, Some _) -> impossible "not accepted by parser" | Newtypes [] -> impossible "not accepted by parser" | Newtypes names -> @@ -1321,7 +1449,7 @@ and fmt_fun_args c args = the first returned value belongs to a box of level N. *) and fmt_body c ?ext ({ast= body; _} as xbody) = let ctx = Exp body in - let parens = parenze_exp xbody in + let parens = parenze_exp c.conf xbody in match body with | {pexp_desc= Pexp_function cs; pexp_attributes; pexp_loc; _} -> ( ( update_config_maybe_disabled c pexp_loc pexp_attributes @@ -1334,6 +1462,34 @@ and fmt_body c ?ext ({ast= body; _} as xbody) = , update_config_maybe_disabled c pexp_loc pexp_attributes @@ fun c -> fmt_cases c ctx cs $ fmt_if parens ")" $ Cmts.fmt_after c pexp_loc ) + | { pexp_desc= + Pexp_apply + ( { pexp_desc= Pexp_extension ({txt= extension_local; _}, PStr []) + ; _ } + , [(Nolabel, sbody)] ) + ; pexp_loc + ; _ } + when Conf.is_jane_street_local_annotation c.conf "local" + ~test:extension_local + (* Don't wipe away comments before [local_]. *) + && not (Cmts.has_before c.cmts pexp_loc) -> + ( fmt " local_" + , fmt_expression c ~eol:(fmt "@;<1000 0>") (sub_exp c.conf ~ctx sbody) + ) + | { pexp_desc= + Pexp_apply + ( { pexp_desc= Pexp_extension ({txt= extension_exclave; _}, PStr []) + ; _ } + , [(Nolabel, sbody)] ) + ; pexp_loc + ; _ } + when Conf.is_jane_street_local_annotation c.conf "exclave" + ~test:extension_exclave + (* Don't wipe away comments before [exclave_]. *) + && not (Cmts.has_before c.cmts pexp_loc) -> + ( fmt " exclave_" + , fmt_expression c ~eol:(fmt "@;<1000 0>") (sub_exp c.conf ~ctx sbody) + ) | _ -> (noop, fmt_expression c ~eol:(fmt "@;<1000 0>") xbody) and fmt_indexop_access c ctx ~fmt_atrs ~has_attr ~parens x = @@ -1348,22 +1504,24 @@ and fmt_indexop_access c ctx ~fmt_atrs ~has_attr ~parens x = Params.parens_if parens c.conf (hovbox 0 ( Params.parens_if inner_wrap c.conf - ( fmt_expression c (sub_exp ~ctx pia_lhs) + ( fmt_expression c (sub_exp c.conf ~ctx pia_lhs) $ str "." $ ( match pia_kind with | Builtin idx -> - wrap_paren (fmt_expression c (sub_exp ~ctx idx)) + wrap_paren (fmt_expression c (sub_exp c.conf ~ctx idx)) | Dotop (path, op, [idx]) -> opt path (fun x -> fmt_longident_loc c x $ str ".") $ str op - $ wrap_paren (fmt_expression c (sub_exp ~ctx idx)) + $ wrap_paren (fmt_expression c (sub_exp c.conf ~ctx idx)) | Dotop (path, op, idx) -> opt path (fun x -> fmt_longident_loc c x $ str ".") $ str op $ wrap_paren - (list idx ";@ " (sub_exp ~ctx >> fmt_expression c)) ) + (list idx ";@ " + (sub_exp c.conf ~ctx >> fmt_expression c) ) ) $ opt pia_rhs (fun e -> - fmt_assign_arrow c $ fmt_expression c (sub_exp ~ctx e) ) ) + fmt_assign_arrow c + $ fmt_expression c (sub_exp c.conf ~ctx e) ) ) $ fmt_atrs ) ) (** Format [Pexp_fun] or [Pexp_newtype]. [wrap_intro] wraps up to after the @@ -1381,7 +1539,7 @@ and fmt_fun ?force_closing_paren let cmts = Cmts.fmt_before ?eol c ast.pexp_loc in if has_label then (false, noop, cmts) else (has_cmts, cmts, noop) in - let xargs, xbody = Sugar.fun_ c.cmts xast in + let xargs, xbody = Sugar.fun_ c.conf c.cmts xast in let fmt_cstr, xbody = type_constr_and_body c xbody in let body = let box = @@ -1457,7 +1615,7 @@ and expression_width c xe = and fmt_args_grouped ?epi:(global_epi = noop) c ctx args = let fmt_arg c ~first:_ ~last (lbl, arg) = - let ({ast; _} as xarg) = sub_exp ~ctx arg in + let ({ast; _} as xarg) = sub_exp c.conf ~ctx arg in let box = match ast.pexp_desc with | Pexp_fun _ | Pexp_function _ -> Some false @@ -1481,7 +1639,7 @@ and fmt_args_grouped ?epi:(global_epi = noop) c ctx args = $ fmt_if_k (not last) (break 1 0) in let is_simple (lbl, x) = - let xexp = sub_exp ~ctx x in + let xexp = sub_exp c.conf ~ctx x in let output = Cmts.preserve ~cache_key:(Arg (lbl, x)) @@ -1527,7 +1685,7 @@ and fmt_sequence c ?ext ~has_attr parens width xexp fmt_atrs = let break (_, xexp1) (_, xexp2) = not (is_simple xexp1 && is_simple xexp2) in - let elts = Sugar.sequence c.cmts xexp in + let elts = Sugar.sequence c.conf c.cmts xexp in ( match elts with | (None, _) :: (first_ext, _) :: _ -> let compare {txt= x; _} {txt= y; _} = String.compare x y in @@ -1596,8 +1754,8 @@ and fmt_infix_op_args c ~parens xexp op_args = in let fmt_arg ~pro ~very_last xarg = let parens = - ((not very_last) && exposed_right_exp Ast.Non_apply xarg.ast) - || parenze_exp xarg + ((not very_last) && exposed_right_exp c.conf Ast.Non_apply xarg.ast) + || parenze_exp c.conf xarg in if Params.Exp.Infix_op_arg.dock c.conf xarg then (* Indentation of docked fun or function start before the operator. *) @@ -1676,10 +1834,18 @@ and fmt_match c ?pro ~parens ?ext ctx xexp cs e0 keyword = $ fmt_extension_suffix c ext $ fmt_attributes c xexp.ast.pexp_attributes $ fmt "@;<1 2>" - $ fmt_expression c (sub_exp ~ctx e0) + $ fmt_expression c (sub_exp c.conf ~ctx e0) $ fmt "@ with" ) $ fmt "@ " $ fmt_cases c ctx cs ) ) +and maybe_fmt_expression_extension c ~pexp_loc ~fmt_atrs ~has_attr ~parens + ~ctx exp fmt_normal_expr = + match Extensions.Expression.of_ast exp with + | Some eexp -> + fmt_expression_extension c ~pexp_loc ~fmt_atrs ~has_attr ~parens ~ctx + eexp + | None -> fmt_normal_expr () + and fmt_expression c ?(box = true) ?(pro = noop) ?eol ?parens ?(indent_wrap = 0) ?ext ({ast= exp; ctx= ctx0} as xexp) = protect c (Exp exp) @@ -1691,14 +1857,16 @@ and fmt_expression c ?(box = true) ?(pro = noop) ?eol ?parens let fmt_cmts = Cmts.fmt c ?eol pexp_loc in let fmt_atrs = fmt_attributes c ~pre:Space pexp_attributes in let has_attr = not (List.is_empty pexp_attributes) in - let parens = Option.value parens ~default:(parenze_exp xexp) in + let parens = Option.value parens ~default:(parenze_exp c.conf xexp) in let ctx = Exp exp in let fmt_args_grouped ?epi e0 a1N = fmt_args_grouped c ctx ?epi ((Nolabel, e0) :: a1N) in hvbox_if box 0 ~name:"expr" @@ fmt_cmts - @@ + @@ maybe_fmt_expression_extension c ~pexp_loc ~fmt_atrs ~has_attr ~parens + ~ctx exp + @@ fun () -> match pexp_desc with | Pexp_apply (_, []) -> impossible "not produced by parser" | Pexp_sequence @@ -1711,13 +1879,16 @@ and fmt_expression c ?(box = true) ?(pro = noop) ?eol ?parens ; pstr_loc= _ } as pld ) ] ) ; _ } , e2 ) -> - let xargs, xbody = Sugar.fun_ c.cmts (sub_exp ~ctx:(Str pld) call) in + let xargs, xbody = + Sugar.fun_ c.conf c.cmts (sub_exp c.conf ~ctx:(Str pld) call) + in let fmt_cstr, xbody = type_constr_and_body c xbody in let is_simple x = is_simple c.conf (expression_width c) x in let break xexp1 xexp2 = not (is_simple xexp1 && is_simple xexp2) in let grps = List.group - (List.map ~f:snd (Sugar.sequence c.cmts (sub_exp ~ctx e2))) + (List.map ~f:snd + (Sugar.sequence c.conf c.cmts (sub_exp c.conf ~ctx e2)) ) ~break in let fmt_grp grp = list grp " ;@ " (fmt_expression c) in @@ -1746,12 +1917,14 @@ and fmt_expression c ?(box = true) ?(pro = noop) ?eol ?parens Pstr_eval (({pexp_desc= Pexp_fun _; _} as retn), []) ; pstr_loc= _ } as pld ) ] ) ; _ } ) -> - let xargs, xbody = Sugar.fun_ c.cmts (sub_exp ~ctx:(Str pld) retn) in + let xargs, xbody = + Sugar.fun_ c.conf c.cmts (sub_exp c.conf ~ctx:(Str pld) retn) + in let fmt_cstr, xbody = type_constr_and_body c xbody in pro $ hvbox 0 (Params.Exp.wrap c.conf ~parens - ( fmt_expression c (sub_exp ~ctx e0) + ( fmt_expression c (sub_exp c.conf ~ctx e0) $ fmt "@\n" $ Cmts.fmt c loc (fmt "|>@\n") $ hvbox c.conf.fmt_opts.extension_indent.v @@ -1764,7 +1937,7 @@ and fmt_expression c ?(box = true) ?(pro = noop) ?eol ?parens $ fmt "@ ->" ) $ fmt "@ " $ fmt_expression c xbody ) ) ) ) | Pexp_infix ({txt= ":="; loc}, r, v) - when is_simple c.conf (expression_width c) (sub_exp ~ctx r) -> + when is_simple c.conf (expression_width c) (sub_exp c.conf ~ctx r) -> let bol_indent = Params.Indent.assignment_operator_bol c.conf in let cmts_before = let indent, adj = @@ -1782,15 +1955,18 @@ and fmt_expression c ?(box = true) ?(pro = noop) ?eol ?parens (hovbox 0 ( match c.conf.fmt_opts.assignment_operator.v with | `Begin_line -> - hvbox 0 (fmt_expression c (sub_exp ~ctx r) $ cmts_before) + hvbox 0 + (fmt_expression c (sub_exp c.conf ~ctx r) $ cmts_before) $ break 1 bol_indent $ fmt ":= " $ cmts_after - $ hvbox 2 (fmt_expression c (sub_exp ~ctx v)) + $ hvbox 2 (fmt_expression c (sub_exp c.conf ~ctx v)) | `End_line -> hvbox 0 - ( hvbox 0 (fmt_expression c (sub_exp ~ctx r) $ cmts_before) + ( hvbox 0 + ( fmt_expression c (sub_exp c.conf ~ctx r) + $ cmts_before ) $ str " :=" ) $ fmt "@;<1 2>" $ cmts_after - $ hvbox 2 (fmt_expression c (sub_exp ~ctx v)) ) ) + $ hvbox 2 (fmt_expression c (sub_exp c.conf ~ctx v)) ) ) | Pexp_prefix ({txt= ("~-" | "~-." | "~+" | "~+.") as op; loc}, e1) -> let op = if Location.width loc = String.length op - 1 then @@ -1801,24 +1977,25 @@ and fmt_expression c ?(box = true) ?(pro = noop) ?eol ?parens pro $ Params.parens_if parens c.conf ( Cmts.fmt c pexp_loc - @@ hvbox 2 (str op $ spc $ fmt_expression c (sub_exp ~ctx e1)) + @@ hvbox 2 + (str op $ spc $ fmt_expression c (sub_exp c.conf ~ctx e1)) $ fmt_atrs ) | Pexp_infix (({txt= id; _} as op), l, ({pexp_desc= Pexp_ident _; _} as r)) when Std_longident.String_id.is_hash_getter id -> pro $ Params.parens_if parens c.conf - ( fmt_expression c (sub_exp ~ctx l) + ( fmt_expression c (sub_exp c.conf ~ctx l) $ hvbox 0 (fmt_str_loc c op) - $ fmt_expression c (sub_exp ~ctx r) ) + $ fmt_expression c (sub_exp c.conf ~ctx r) ) | Pexp_infix (op, l, ({pexp_desc= Pexp_fun _; pexp_loc; pexp_attributes; _} as r)) when not c.conf.fmt_opts.break_infix_before_func.v -> (* side effects of Cmts.fmt c.cmts before Sugar.fun_ is important *) let cmts_before = Cmts.fmt_before c pexp_loc in let cmts_after = Cmts.fmt_after c pexp_loc in - let xr = sub_exp ~ctx r in - let parens_r = parenze_exp xr in - let xargs, xbody = Sugar.fun_ c.cmts xr in + let xr = sub_exp c.conf ~ctx r in + let parens_r = parenze_exp c.conf xr in + let xargs, xbody = Sugar.fun_ c.conf c.cmts xr in let fmt_cstr, xbody = type_constr_and_body c xbody in let indent_wrap = if parens then -2 else 0 in let pre_body, body = fmt_body c ?ext xbody in @@ -1834,7 +2011,8 @@ and fmt_expression c ?(box = true) ?(pro = noop) ?eol ?parens (wrap_if has_attr "(" ")" ( hvbox 2 ( hvbox indent_wrap - ( fmt_expression ~indent_wrap c (sub_exp ~ctx l) + ( fmt_expression ~indent_wrap c + (sub_exp c.conf ~ctx l) $ fmt "@;" $ hovbox 2 ( hvbox 0 @@ -1857,14 +2035,14 @@ and fmt_expression c ?(box = true) ?(pro = noop) ?eol ?parens when not c.conf.fmt_opts.break_infix_before_func.v -> let cmts_before = Cmts.fmt_before c pexp_loc in let cmts_after = Cmts.fmt_after c pexp_loc in - let xr = sub_exp ~ctx r in - let parens_r = parenze_exp xr in + let xr = sub_exp c.conf ~ctx r in + let parens_r = parenze_exp c.conf xr in let indent = Params.Indent.function_ c.conf ~parens xr in pro $ Params.parens_if parens c.conf (hvbox indent ( hvbox 0 - ( fmt_expression c (sub_exp ~ctx l) + ( fmt_expression c (sub_exp c.conf ~ctx l) $ fmt "@;" $ hovbox 2 ( hvbox 0 @@ -1875,7 +2053,9 @@ and fmt_expression c ?(box = true) ?(pro = noop) ?eol ?parens $ fmt "@ " $ fmt_cases c (Exp r) cs $ fmt_if parens_r " )" $ cmts_after ) ) | Pexp_infix _ -> - let op_args = Sugar.Exp.infix c.cmts (prec_ast (Exp exp)) xexp in + let op_args = + Sugar.Exp.infix c.conf c.cmts (prec_ast (Exp exp)) xexp + in let inner_wrap = parens || has_attr in let outer_wrap = match ctx0 with @@ -1929,13 +2109,29 @@ and fmt_expression c ?(box = true) ?(pro = noop) ?eol ?parens (hvbox indent_wrap ( fmt_infix_op_args ~parens:inner_wrap c xexp infix_op_args $ fmt_atrs ) ) ) + | Pexp_apply + ( {pexp_desc= Pexp_extension ({txt= extension_local; _}, PStr []); _} + , [(Nolabel, sbody)] ) + when Conf.is_jane_street_local_annotation c.conf "local" + ~test:extension_local -> + pro + $ Params.parens_if parens c.conf + (fmt "local_@ " $ fmt_expression c (sub_exp c.conf ~ctx sbody)) + | Pexp_apply + ( {pexp_desc= Pexp_extension ({txt= extension_exclave; _}, PStr []); _} + , [(Nolabel, sbody)] ) + when Conf.is_jane_street_local_annotation c.conf "exclave" + ~test:extension_exclave -> + pro + $ Params.parens_if parens c.conf + (fmt "exclave_@ " $ fmt_expression c (sub_exp c.conf ~ctx sbody)) | Pexp_prefix (op, e) -> let has_cmts = Cmts.has_before c.cmts e.pexp_loc in pro $ hvbox 2 (Params.Exp.wrap c.conf ~parens ( fmt_str_loc c op $ fmt_if has_cmts "@," - $ fmt_expression c ~box (sub_exp ~ctx e) + $ fmt_expression c ~box (sub_exp c.conf ~ctx e) $ fmt_atrs ) ) | Pexp_apply (e0, e1N1) -> ( let wrap = @@ -1960,9 +2156,9 @@ and fmt_expression c ?(box = true) ?(pro = noop) ?eol ?parens match last_arg.pexp_desc with | Pexp_fun (_, _, _, eN1_body) when List.for_all args_before ~f:(fun (_, eI) -> - is_simple c.conf (fun _ -> 0) (sub_exp ~ctx eI) ) -> + is_simple c.conf (fun _ -> 0) (sub_exp c.conf ~ctx eI) ) -> (* Last argument is a [fun _ ->]. *) - let xlast_arg = sub_exp ~ctx last_arg in + let xlast_arg = sub_exp c.conf ~ctx last_arg in let args = let break_body = match eN1_body.pexp_desc with @@ -1994,7 +2190,7 @@ and fmt_expression c ?(box = true) ?(pro = noop) ?eol ?parens (expr_epi $ Params.parens_if parens c.conf (args $ fmt_atrs)) | Pexp_function [{pc_lhs; pc_guard= None; pc_rhs}] when List.for_all args_before ~f:(fun (_, eI) -> - is_simple c.conf (fun _ -> 0) (sub_exp ~ctx eI) ) -> + is_simple c.conf (fun _ -> 0) (sub_exp c.conf ~ctx eI) ) -> let force = if Location.is_single_line last_arg.pexp_loc @@ -2024,17 +2220,17 @@ and fmt_expression c ?(box = true) ?(pro = noop) ?eol ?parens (sub_pat ~ctx pc_lhs) $ fmt "@ ->" ) $ fmt "@ " - $ cbox 0 (fmt_expression c (sub_exp ~ctx pc_rhs)) + $ cbox 0 (fmt_expression c (sub_exp c.conf ~ctx pc_rhs)) $ closing_paren c ~force $ Cmts.fmt_after c last_arg.pexp_loc ) $ fmt_atrs ) ) | Pexp_function cs when List.for_all args_before ~f:(fun (_, eI) -> - is_simple c.conf (fun _ -> 0) (sub_exp ~ctx eI) ) -> + is_simple c.conf (fun _ -> 0) (sub_exp c.conf ~ctx eI) ) -> let wrap = if c.conf.fmt_opts.wrap_fun_args.v then hovbox 2 else hvbox 2 in - let xlast_arg = sub_exp ~ctx last_arg in + let xlast_arg = sub_exp c.conf ~ctx last_arg in let ctx'' = Exp last_arg in hvbox (Params.Indent.docked_function c.conf ~parens xlast_arg) @@ -2076,7 +2272,7 @@ and fmt_expression c ?(box = true) ?(pro = noop) ?eol ?parens (Params.parens_if parens c.conf ( p.box (fmt_expressions c (expression_width c) (sub_exp ~ctx) e1N - (sub_exp ~ctx >> fmt_expression c) + (sub_exp c.conf ~ctx >> fmt_expression c) p pexp_loc ) $ fmt_atrs ) ) | Pexp_list e1N -> @@ -2092,14 +2288,15 @@ and fmt_expression c ?(box = true) ?(pro = noop) ?eol ?parens (fmt_expressions c (expression_width c) (sub_exp ~ctx) e1N (fun e -> let fmt_cmts = Cmts.fmt c ~eol:cmt_break e.pexp_loc in - fmt_cmts @@ (sub_exp ~ctx >> fmt_expression c) e ) + fmt_cmts @@ (sub_exp c.conf ~ctx >> fmt_expression c) e + ) p pexp_loc ) $ fmt_atrs ) ) | Pexp_assert e0 -> let paren_body = if Exp.is_symbol e0 || Exp.is_monadic_binding e0 then not (List.is_empty e0.pexp_attributes) - else parenze_exp (sub_exp ~ctx e0) + else parenze_exp c.conf (sub_exp c.conf ~ctx e0) in pro $ hovbox 0 @@ -2109,7 +2306,8 @@ and fmt_expression c ?(box = true) ?(pro = noop) ?eol ?parens ( str "assert" $ fmt_extension_suffix c ext $ fmt_or paren_body " (@," "@ " - $ fmt_expression c ~parens:false (sub_exp ~ctx e0) ) + $ fmt_expression c ~parens:false (sub_exp c.conf ~ctx e0) + ) $ fmt_if_k paren_body (closing_paren c) $ fmt_atrs ) ) ) | Pexp_constant const -> @@ -2123,7 +2321,7 @@ and fmt_expression c ?(box = true) ?(pro = noop) ?eol ?parens $ hvbox (Params.Indent.exp_constraint c.conf) ( wrap_fits_breaks ~space:false c.conf "(" ")" - ( fmt_expression c (sub_exp ~ctx e) + ( fmt_expression c (sub_exp c.conf ~ctx e) $ fmt "@ : " $ fmt_core_type c (sub_typ ~ctx t) ) $ fmt_atrs ) @@ -2145,31 +2343,32 @@ and fmt_expression c ?(box = true) ?(pro = noop) ?eol ?parens ( hvbox indent_wrap (fmt_infix_op_args c ~parens xexp (List.mapi l ~f:(fun i e -> - (None, None, (fmt_if (i > 0) "::", sub_exp ~ctx e)) ) - ) ) + ( None + , None + , (fmt_if (i > 0) "::", sub_exp c.conf ~ctx e) ) ) ) ) $ fmt_atrs ) | Pexp_construct (lid, Some arg) -> pro $ Params.parens_if parens c.conf ( hvbox 2 ( fmt_longident_loc c lid $ fmt "@ " - $ fmt_expression c (sub_exp ~ctx arg) ) + $ fmt_expression c (sub_exp c.conf ~ctx arg) ) $ fmt_atrs ) | Pexp_variant (s, arg) -> pro $ hvbox 2 (Params.parens_if parens c.conf ( variant_var c s - $ opt arg (fmt "@ " >$ (sub_exp ~ctx >> fmt_expression c)) + $ opt arg (fmt "@ " >$ (sub_exp c.conf ~ctx >> fmt_expression c)) $ fmt_atrs ) ) | Pexp_field (exp, lid) -> pro $ hvbox 2 (Params.parens_if parens c.conf - ( fmt_expression c (sub_exp ~ctx exp) + ( fmt_expression c (sub_exp c.conf ~ctx exp) $ fmt "@,." $ fmt_longident_loc c lid $ fmt_atrs ) ) | Pexp_newtype _ | Pexp_fun _ -> - let xargs, xbody = Sugar.fun_ c.cmts xexp in + let xargs, xbody = Sugar.fun_ c.conf c.cmts xexp in let fmt_cstr, xbody = type_constr_and_body c xbody in let body_is_function = match xbody.ast.pexp_desc with Pexp_function _ -> true | _ -> false @@ -2227,13 +2426,14 @@ and fmt_expression c ?(box = true) ?(pro = noop) ?eol ?parens let cnd_exps = let with_conds = List.map if_branches ~f:(fun x -> - ( Some (sub_exp ~ctx x.if_cond) - , sub_exp ~ctx x.if_body + ( Some (sub_exp c.conf ~ctx x.if_cond) + , sub_exp c.conf ~ctx x.if_body , x.if_attrs ) ) in match else_ with | Some x -> - List.rev ((None, sub_exp ~ctx x, []) :: List.rev with_conds) + List.rev + ((None, sub_exp c.conf ~ctx x, []) :: List.rev with_conds) | None -> with_conds in pro @@ -2244,7 +2444,7 @@ and fmt_expression c ?(box = true) ?(pro = noop) ?eol ?parens (fun ~first ~last (xcond, xbch, pexp_attributes) -> let symbol_parens = Exp.is_symbol xbch.ast in let parens_bch = - parenze_exp xbch && not symbol_parens + parenze_exp c.conf xbch && not symbol_parens in let parens_exp = false in let p = @@ -2272,16 +2472,19 @@ and fmt_expression c ?(box = true) ?(pro = noop) ?eol ?parens $ fmt_atrs ) | Pexp_let (lbs, body) -> let bindings = - Sugar.Let_binding.of_let_bindings c.cmts ~ctx lbs.pvbs_bindings + Sugar.Let_binding.of_let_bindings c.conf c.cmts ~ctx + lbs.pvbs_bindings in - let fmt_expr = fmt_expression c (sub_exp ~ctx body) in + let fmt_expr = fmt_expression c (sub_exp c.conf ~ctx body) in let ext = lbs.pvbs_extension in pro $ fmt_let_bindings c ?ext ~parens ~fmt_atrs ~fmt_expr ~has_attr lbs.pvbs_rec bindings body | Pexp_letop {let_; ands; body} -> - let bd = Sugar.Let_binding.of_binding_ops c.cmts ~ctx (let_ :: ands) in - let fmt_expr = fmt_expression c (sub_exp ~ctx body) in + let bd = + Sugar.Let_binding.of_binding_ops c.conf c.cmts ~ctx (let_ :: ands) + in + let fmt_expr = fmt_expression c (sub_exp c.conf ~ctx body) in pro $ fmt_let_bindings c ?ext ~parens ~fmt_atrs ~fmt_expr ~has_attr Nonrecursive bd body @@ -2300,7 +2503,7 @@ and fmt_expression c ?(box = true) ?(pro = noop) ?eol ?parens (pre $ fmt_extension_constructor c ctx ext_cstr) ) $ fmt "@ in" ) $ fmt "@;<1000 0>" - $ fmt_expression c (sub_exp ~ctx exp) ) + $ fmt_expression c (sub_exp c.conf ~ctx exp) ) $ fmt_atrs ) | Pexp_letmodule (name, args, pmod, exp) -> let keyword = "let module" in @@ -2331,20 +2534,30 @@ and fmt_expression c ?(box = true) ?(pro = noop) ?eol ?parens ~attrs:(Ast_helper.Attr.ext_attrs ?ext ()) ~epi:(str "in") ~can_sparse ~rec_flag:false ) $ fmt "@;<1000 0>" - $ fmt_expression c (sub_exp ~ctx exp) ) + $ fmt_expression c (sub_exp c.conf ~ctx exp) ) $ fmt_atrs ) | Pexp_open (lid, e0) -> + let can_skip_parens_extension attrs : Extensions.Expression.t -> _ = + function + | Eexp_comprehension + (Cexp_list_comprehension _ | Cexp_array_comprehension _) + |Eexp_immutable_array (Iaexp_immutable_array _) -> + List.is_empty attrs + in let can_skip_parens = (not (Cmts.has_before c.cmts e0.pexp_loc)) && (not (Cmts.has_after c.cmts e0.pexp_loc)) && - match e0.pexp_desc with - | (Pexp_array _ | Pexp_list _ | Pexp_record _) - when List.is_empty e0.pexp_attributes -> - true - | Pexp_tuple _ -> Poly.(c.conf.fmt_opts.parens_tuple.v = `Always) - | Pexp_construct ({txt= Lident "[]"; _}, None) -> true - | _ -> false + match Extensions.Expression.of_ast e0 with + | Some ee0 -> can_skip_parens_extension e0.pexp_attributes ee0 + | None -> ( + match e0.pexp_desc with + | (Pexp_array _ | Pexp_list _ | Pexp_record _) + when List.is_empty e0.pexp_attributes -> + true + | Pexp_tuple _ -> Poly.(c.conf.fmt_opts.parens_tuple.v = `Always) + | Pexp_construct ({txt= Lident "[]"; _}, None) -> true + | _ -> false ) in let outer_parens = has_attr && parens in let inner_parens = not can_skip_parens in @@ -2356,7 +2569,7 @@ and fmt_expression c ?(box = true) ?(pro = noop) ?eol ?parens ( fmt_longident_loc c lid $ str "." $ fmt_if inner_parens "(" ) $ fmt "@;<0 2>" - $ fmt_expression c (sub_exp ~ctx e0) + $ fmt_expression c (sub_exp c.conf ~ctx e0) $ fmt_if_k inner_parens (closing_paren c) ) $ fmt_atrs ) ) | Pexp_letopen @@ -2389,7 +2602,7 @@ and fmt_expression c ?(box = true) ?(pro = noop) ?eol ?parens $ Cmts.fmt_after c popen_loc $ str " in" ) $ break 1000 0 - $ fmt_expression c (sub_exp ~ctx e0) ) ) ) + $ fmt_expression c (sub_exp c.conf ~ctx e0) ) ) ) $ fmt_atrs ) ) | Pexp_try (e0, [{pc_lhs; pc_guard; pc_rhs}]) when Poly.( @@ -2397,11 +2610,11 @@ and fmt_expression c ?(box = true) ?(pro = noop) ?eol ?parens && c.conf.fmt_opts.break_cases.v <> `All && c.conf.fmt_opts.break_cases.v <> `Vertical ) -> (* side effects of Cmts.fmt_before before [fmt_pattern] is important *) - let xpc_rhs = sub_exp ~ctx pc_rhs in + let xpc_rhs = sub_exp c.conf ~ctx pc_rhs in let leading_cmt = Cmts.fmt_before c pc_lhs.ppat_loc in let parens_here, parens_for_exp = if c.conf.fmt_opts.leading_nested_match_parens.v then (false, None) - else (parenze_exp xpc_rhs, Some false) + else (parenze_exp c.conf xpc_rhs, Some false) in pro $ Params.Exp.wrap c.conf ~parens ~disambiguate:true @@ -2411,7 +2624,7 @@ and fmt_expression c ?(box = true) ?(pro = noop) ?eol ?parens $ fmt_extension_suffix c ext $ fmt_attributes c pexp_attributes $ fmt "@;<1 2>" - $ fmt_expression c (sub_exp ~ctx e0) ) + $ fmt_expression c (sub_exp c.conf ~ctx e0) ) $ break 1 (-2) $ hvbox 0 ( hvbox 0 @@ -2421,7 +2634,7 @@ and fmt_expression c ?(box = true) ?(pro = noop) ?eol ?parens (sub_pat ~ctx pc_lhs) $ opt pc_guard (fun g -> fmt "@ when " - $ fmt_expression c (sub_exp ~ctx g) ) + $ fmt_expression c (sub_exp c.conf ~ctx g) ) $ fmt "@ ->" $ fmt_if parens_here " (" ) ) $ fmt "@;<1 2>" $ cbox 0 (fmt_expression c ?parens:parens_for_exp xpc_rhs) @@ -2473,7 +2686,8 @@ and fmt_expression c ?(box = true) ?(pro = noop) ?eol ?parens let typ1 = Option.map typ1 ~f:(sub_typ ~ctx) in let typ2 = Option.map typ2 ~f:(sub_typ ~ctx) in let rhs = - Option.map exp ~f:(fun e -> fmt_expression c (sub_exp ~ctx e)) + Option.map exp ~f:(fun e -> + fmt_expression c (sub_exp c.conf ~ctx e) ) in hvbox 0 @@ fmt_record_field c ?typ1 ?typ2 ?rhs lid in @@ -2494,7 +2708,8 @@ and fmt_expression c ?(box = true) ?(pro = noop) ?eol ?parens ( p1.box ( opt default (fun d -> hvbox 2 - (fmt_expression c (sub_exp ~ctx d) $ fmt "@;<1 -2>") + ( fmt_expression c (sub_exp c.conf ~ctx d) + $ fmt "@;<1 -2>" ) $ str "with" $ p2.break_after_with ) $ fmt_fields ) $ fmt_atrs ) @@ -2508,7 +2723,7 @@ and fmt_expression c ?(box = true) ?(pro = noop) ?eol ?parens , _ ) ; pstr_loc= _ } ] ) when Source.extension_using_sugar ~name:ext ~payload:e1.pexp_loc - && List.length (Sugar.sequence c.cmts xexp) > 1 -> + && List.length (Sugar.sequence c.conf c.cmts xexp) > 1 -> pro $ fmt_sequence ~has_attr c parens (expression_width c) xexp fmt_atrs ~ext @@ -2521,9 +2736,9 @@ and fmt_expression c ?(box = true) ?(pro = noop) ?eol ?parens $ hvbox 0 (Params.Exp.wrap c.conf ~parens ( Params.parens_if has_attr c.conf - ( fmt_expression c (sub_exp ~ctx e1) + ( fmt_expression c (sub_exp c.conf ~ctx e1) $ str "." $ fmt_longident_loc c lid $ fmt_assign_arrow c - $ fmt_expression c (sub_exp ~ctx e2) ) + $ fmt_expression c (sub_exp c.conf ~ctx e2) ) $ fmt_atrs ) ) | Pexp_tuple es -> let parens = @@ -2555,7 +2770,7 @@ and fmt_expression c ?(box = true) ?(pro = noop) ?eol ?parens (Params.wrap_tuple ~parens:inner_wrap ~no_parens_if_break c.conf (list es (Params.comma_sep c.conf) - (sub_exp ~ctx >> fmt_expression c) ) ) + (sub_exp c.conf ~ctx >> fmt_expression c) ) ) $ fmt_atrs ) ) | Pexp_lazy e -> pro @@ -2564,7 +2779,7 @@ and fmt_expression c ?(box = true) ?(pro = noop) ?eol ?parens ( str "lazy" $ fmt_extension_suffix c ext $ fmt "@ " - $ fmt_expression c (sub_exp ~ctx e) + $ fmt_expression c (sub_exp c.conf ~ctx e) $ fmt_atrs ) ) | Pexp_extension ( ext @@ -2590,7 +2805,7 @@ and fmt_expression c ?(box = true) ?(pro = noop) ?eol ?parens $ hvbox 0 (Params.parens_if outer_parens c.conf ( fmt_expression c ~box ?eol ~parens:inner_parens ~ext - (sub_exp ~ctx:(Str str) e1) + (sub_exp c.conf ~ctx:(Str str) e1) $ fmt_atrs ) ) | Pexp_extension ( ext @@ -2606,7 +2821,7 @@ and fmt_expression c ?(box = true) ?(pro = noop) ?eol ?parens pro $ hvbox 0 ( fmt_expression c ~box ?eol ~parens ~ext - (sub_exp ~ctx:(Str str) e1) + (sub_exp c.conf ~ctx:(Str str) e1) $ fmt_atrs ) | Pexp_extension ext -> pro @@ -2628,12 +2843,12 @@ and fmt_expression c ?(box = true) ?(pro = noop) ?eol ?parens $ hovbox 0 ( fmt_pattern c (sub_pat ~ctx p1) $ fmt "@ =@;<1 2>" - $ fmt_expression c (sub_exp ~ctx e1) + $ fmt_expression c (sub_exp c.conf ~ctx e1) $ fmt_direction_flag dir - $ fmt_expression c (sub_exp ~ctx e2) ) + $ fmt_expression c (sub_exp c.conf ~ctx e2) ) $ fmt "@;do" ) $ fmt "@;<1000 0>" - $ fmt_expression c (sub_exp ~ctx e3) ) + $ fmt_expression c (sub_exp c.conf ~ctx e3) ) $ fmt "@;<1000 0>done" ) $ fmt_atrs ) ) | Pexp_coerce (e1, t1, t2) -> @@ -2641,7 +2856,7 @@ and fmt_expression c ?(box = true) ?(pro = noop) ?eol ?parens $ hvbox 2 (Params.parens_if (parens && has_attr) c.conf ( wrap_fits_breaks ~space:false c.conf "(" ")" - ( fmt_expression c (sub_exp ~ctx e1) + ( fmt_expression c (sub_exp c.conf ~ctx e1) $ opt t1 (fmt "@ : " >$ (sub_typ ~ctx >> fmt_core_type c)) $ fmt "@ :> " $ fmt_core_type c (sub_typ ~ctx t2) ) @@ -2656,10 +2871,10 @@ and fmt_expression c ?(box = true) ?(pro = noop) ?eol ?parens ( str "while" $ fmt_extension_suffix c ext $ fmt "@;<1 2>" - $ fmt_expression c (sub_exp ~ctx e1) + $ fmt_expression c (sub_exp c.conf ~ctx e1) $ fmt "@;do" ) $ fmt "@;<1000 0>" - $ fmt_expression c (sub_exp ~ctx e2) ) + $ fmt_expression c (sub_exp c.conf ~ctx e2) ) $ fmt "@;<1000 0>done" ) $ fmt_atrs ) ) | Pexp_unreachable -> pro $ str "." @@ -2667,7 +2882,7 @@ and fmt_expression c ?(box = true) ?(pro = noop) ?eol ?parens pro $ hvbox 2 (Params.parens_if parens c.conf - ( fmt_expression c (sub_exp ~ctx exp) + ( fmt_expression c (sub_exp c.conf ~ctx exp) $ fmt "@,#" $ fmt_str_loc c meth $ fmt_atrs ) ) | Pexp_new {txt; loc} -> pro @@ -2695,7 +2910,7 @@ and fmt_expression c ?(box = true) ?(pro = noop) ?eol ?parens | _ -> Cmts.fmt c ~eol loc @@ fmt_longident txt $ str " = " - $ fmt_expression c (sub_exp ~ctx f) + $ fmt_expression c (sub_exp c.conf ~ctx f) in match l with | [] -> @@ -2715,7 +2930,7 @@ and fmt_expression c ?(box = true) ?(pro = noop) ?eol ?parens (Params.Exp.wrap c.conf ~parens ( Params.parens_if has_attr c.conf ( fmt_str_loc c name $ fmt_assign_arrow c - $ hvbox 2 (fmt_expression c (sub_exp ~ctx expr)) ) + $ hvbox 2 (fmt_expression c (sub_exp c.conf ~ctx expr)) ) $ fmt_atrs ) ) | Pexp_indexop_access x -> pro $ fmt_indexop_access c ctx ~fmt_atrs ~has_attr ~parens x @@ -2732,14 +2947,86 @@ and fmt_expression c ?(box = true) ?(pro = noop) ?eol ?parens pro $ wrap_beginend (fmt_expression c ~box ?eol ~parens:false ~indent_wrap ?ext - (sub_exp ~ctx e) ) + (sub_exp c.conf ~ctx e) ) | Pexp_parens e -> pro $ hvbox 0 (fmt_expression c ~box ?eol ~parens:true ~indent_wrap ?ext - (sub_exp ~ctx e) ) + (sub_exp c.conf ~ctx e) ) $ fmt_atrs +and fmt_expression_extension c ~pexp_loc ~fmt_atrs ~has_attr ~parens ~ctx : + Extensions.Expression.t -> _ = function + | Eexp_comprehension cexpr -> + let punctuation, space_around, comp = + match cexpr with + | Cexp_list_comprehension comp -> + ("", c.conf.fmt_opts.space_around_lists.v, comp) + | Cexp_array_comprehension (amut, comp) -> + let punct = + match amut with Mutable _ -> "|" | Immutable -> ":" + in + (punct, c.conf.fmt_opts.space_around_arrays.v, comp) + in + hvbox_if has_attr 0 + (Params.parens_if parens c.conf + ( Params.wrap_comprehension c.conf ~space_around ~punctuation + (fmt_comprehension c ~ctx comp) + $ fmt_atrs ) ) + | Eexp_immutable_array (Iaexp_immutable_array []) -> + hvbox 0 + (Params.parens_if parens c.conf + ( wrap_fits_breaks c.conf "[:" ":]" (Cmts.fmt_within c pexp_loc) + $ fmt_atrs ) ) + | Eexp_immutable_array (Iaexp_immutable_array e1N) -> + let p = Params.get_iarray_expr c.conf in + hvbox_if has_attr 0 + (Params.parens_if parens c.conf + ( p.box + (fmt_expressions c (expression_width c) (sub_exp ~ctx) e1N + (sub_exp c.conf ~ctx >> fmt_expression c) + p pexp_loc ) + $ fmt_atrs ) ) + +and fmt_comprehension c ~ctx Extensions.Comprehensions.{body; clauses} = + hvbox 0 (* Don't indent the clauses to the right of the body *) + ( fmt_expression c (sub_exp c.conf ~ctx body) + $ sequence (List.map clauses ~f:(fmt_comprehension_clause ~ctx c)) ) + +and fmt_comprehension_clause c ~ctx + (clause : Extensions.Comprehensions.clause) = + let subclause kwd formatter item = + fits_breaks " " ~hint:(1000, 0) "" + $ hvbox 2 (str kwd $ sp (Break (1, 0)) $ hovbox 2 (formatter c item)) + in + match clause with + | For bindings -> + list_fl bindings (fun ~first ~last:_ -> + subclause + (if first then "for" else "and") + (fmt_comprehension_binding ~ctx) ) + | When cond -> + subclause "when" + (fun c xt -> fmt_expression c xt) + (sub_exp c.conf ~ctx cond) + +and fmt_comprehension_binding c ~ctx + Extensions.Comprehensions.{pattern; iterator; attributes} = + fmt_attributes c attributes + $ fmt_pattern c (sub_pat ~ctx pattern) + $ sp Space + (* The harder break is after the [=]/[in] *) + $ fmt_comprehension_iterator c ~ctx iterator + +and fmt_comprehension_iterator c ~ctx : + Extensions.Comprehensions.iterator -> _ = function + | Range {start; stop; direction} -> + fmt "=@;<1 0>" + $ fmt_expression c (sub_exp c.conf ~ctx start) + $ fmt_direction_flag direction + $ fmt_expression c (sub_exp c.conf ~ctx stop) + | In seq -> fmt "in@;<1 0>" $ fmt_expression c (sub_exp c.conf ~ctx seq) + and fmt_let_bindings c ?ext ~parens ~has_attr ~fmt_atrs ~fmt_expr rec_flag bindings body = let indent_after_in = @@ -2853,6 +3140,7 @@ and fmt_class_type ?(pro = noop) c ({ast= typ; _} as xtyp) = | Pcty_arrow (args, rhs) -> Cmts.relocate c.cmts ~src:pcty_loc ~before:(List.hd_exn args).pap_type.ptyp_loc ~after:rhs.pcty_loc ; + let args = List.map ~f:(fun arg -> (arg, false)) args in let pro = pro ~cmt:true $ fmt_arrow_type c ~ctx ~parens:false ~parent_has_parens:parens args @@ -2878,7 +3166,7 @@ and fmt_class_expr c ({ast= exp; ctx= ctx0} as xexp) = let {pcl_desc; pcl_loc; pcl_attributes} = exp in update_config_maybe_disabled c pcl_loc pcl_attributes @@ fun c -> - let parens = parenze_cl xexp in + let parens = parenze_cl c.conf xexp in let ctx = Cl exp in let fmt_args_grouped e0 a1N = (* TODO: consider [e0] when grouping *) @@ -2898,7 +3186,7 @@ and fmt_class_expr c ({ast= exp; ctx= ctx0} as xexp) = ( fmt_class_structure c ~ctx ?ext:None pcstr_self pcstr_fields $ fmt_atrs ) ) | Pcl_fun _ -> - let xargs, xbody = Sugar.cl_fun c.cmts xexp in + let xargs, xbody = Sugar.cl_fun c.conf c.cmts xexp in let indent = match ctx0 with | Cl {pcl_desc= Pcl_fun _; _} -> 0 @@ -2925,7 +3213,8 @@ and fmt_class_expr c ({ast= exp; ctx= ctx0} as xexp) = | _ -> c.conf.fmt_opts.indent_after_in.v in let bindings = - Sugar.Let_binding.of_let_bindings c.cmts ~ctx lbs.pvbs_bindings + Sugar.Let_binding.of_let_bindings c.conf c.cmts ~ctx + lbs.pvbs_bindings in let fmt_expr = fmt_class_expr c (sub_cl ~ctx body) in let has_attr = not (List.is_empty pcl_attributes) in @@ -2977,26 +3266,26 @@ and fmt_class_field_kind c ctx = function $ fmt_core_type ~pro:"." ~pro_space:false c (sub_typ ~ctx t) , noop , fmt "@;<1 2>=" - , fmt "@ " $ fmt_expression c (sub_exp ~ctx e) ) + , fmt "@ " $ fmt_expression c (sub_exp c.conf ~ctx e) ) | None -> ( fmt "@ : " $ fmt_core_type c (sub_typ ~ctx poly) , noop , fmt "@;<1 2>=" - , fmt "@ " $ fmt_expression c (sub_exp ~ctx e) ) ) + , fmt "@ " $ fmt_expression c (sub_exp c.conf ~ctx e) ) ) | Cfk_concrete (_, {pexp_desc= Pexp_poly (e, poly); pexp_loc; _}) -> let xargs, xbody = match poly with | None -> - Sugar.fun_ c.cmts ~will_keep_first_ast_node:false - (sub_exp ~ctx e) - | Some _ -> ([], sub_exp ~ctx e) + Sugar.fun_ c.conf c.cmts ~will_keep_first_ast_node:false + (sub_exp c.conf ~ctx e) + | Some _ -> ([], sub_exp c.conf ~ctx e) in let ty, e = match (xbody.ast, poly) with | {pexp_desc= Pexp_constraint (e, t); pexp_loc; _}, None -> Cmts.relocate c.cmts ~src:pexp_loc ~before:t.ptyp_loc ~after:e.pexp_loc ; - (Some t, sub_exp ~ctx e) + (Some t, sub_exp c.conf ~ctx e) | {pexp_desc= Pexp_constraint _; _}, Some _ -> (poly, xbody) | _, poly -> (poly, xbody) in @@ -3017,7 +3306,7 @@ and fmt_class_field_kind c ctx = function ( opt ty (fun t -> fmt "@ : " $ fmt_core_type c (sub_typ ~ctx t)) , noop , fmt "@;<1 2>=" - , fmt "@ " $ fmt_expression c (sub_exp ~ctx e) ) + , fmt "@ " $ fmt_expression c (sub_exp c.conf ~ctx e) ) and fmt_class_field c {ast= cf; _} = protect c (Clf cf) @@ -3074,7 +3363,8 @@ and fmt_class_field c {ast= cf; _} = $ str " = " $ fmt_core_type c (sub_typ ~ctx t2) | Pcf_initializer e -> - str "initializer" $ break 1 2 $ fmt_expression c (sub_exp ~ctx e) + str "initializer" $ break 1 2 + $ fmt_expression c (sub_exp c.conf ~ctx e) | Pcf_attribute attr -> fmt_floating_attributes_and_docstrings c [attr] | Pcf_extension ext -> fmt_item_extension c ctx ext @@ -3125,7 +3415,7 @@ and fmt_cases c ctx cs = list_fl cs (fmt_case c ctx) and fmt_case c ctx ~first ~last case = let {pc_lhs; pc_guard; pc_rhs} = case in - let xrhs = sub_exp ~ctx pc_rhs in + let xrhs = sub_exp c.conf ~ctx pc_rhs in (* side effects of Cmts.fmt_before before [fmt_lhs] is important *) let leading_cmt = Cmts.fmt_before c pc_lhs.ppat_loc in let xlhs = sub_pat ~ctx pc_lhs in @@ -3146,8 +3436,8 @@ and fmt_case c ctx ~first ~last case = ( hvbox 0 ( fmt_pattern c ~pro:p.bar ~parens:paren_lhs xlhs $ opt pc_guard (fun g -> - fmt "@;<1 2>when " $ fmt_expression c (sub_exp ~ctx g) ) - ) + fmt "@;<1 2>when " + $ fmt_expression c (sub_exp c.conf ~ctx g) ) ) $ p.break_before_arrow $ str "->" $ p.break_after_arrow $ p.open_paren_branch ) $ p.break_after_opening_paren @@ -3195,15 +3485,42 @@ and fmt_value_description ?ext c ctx vd = $ fmt_item_attributes c ~pre:(Break (1, 0)) atrs $ doc_after ) +and fmt_tydcl_param c ctx ty = + fmt_core_type ~tydecl_param:true c (sub_typ ~ctx ty) + $ + (* CR layouts v1.5: When we added the syntax for layout annotations on type + parameters to the parser, we also made it possible for people to put + arbitrary attributes on type parameters. Previously, the parser didn't + accept attributes at all here, though there has always been a place in + the parse tree. + + The parser currently allows you to have either a pretty layout + annotation or arbitrary attributes, but not both. A pretty layout + annotation only parses if it's the only attribute, so we only print the + pretty syntax in that case. Probably we'll change this in v1.5. + + In the case of multiple attributes, which may include layouts, they'll + be printed as normal attributes by [fmt_core_type]. So we do nothing + here. *) + match ty.ptyp_attributes with + | [] | _ :: _ :: _ -> noop + | [attr] -> fmt_if_k (is_layout attr) (fmt "@ :@ " $ str attr.attr_name.txt) + and fmt_tydcl_params c ctx params = - fmt_if_k - (not (List.is_empty params)) - ( wrap_fits_breaks_if ~space:false c.conf - (List.length params > 1) - "(" ")" + let empty, parenize = + match params with + | [] -> (true, false) + | [(p, _)] -> + ( false + , match p.ptyp_attributes with + | [] | _ :: _ :: _ -> false + | [attr] -> is_layout attr ) + | _ :: _ :: _ -> (false, true) + in + fmt_if_k (not empty) + ( wrap_fits_breaks_if ~space:false c.conf parenize "(" ")" (list params (Params.comma_sep c.conf) (fun (ty, vc) -> - fmt_variance_injectivity c vc - $ fmt_core_type c (sub_typ ~ctx ty) ) ) + fmt_variance_injectivity c vc $ fmt_tydcl_param c ctx ty ) ) $ fmt "@ " ) and fmt_class_params c ctx params = @@ -3337,6 +3654,7 @@ and fmt_label_declaration c ctx ?(last = false) decl = (fits_breaks ~level:5 "" ";") ) (str ";") in + let is_global, atrs = split_global_flags_from_attrs c.conf atrs in hovbox 0 ( Cmts.fmt_before c pld_loc $ hvbox @@ -3347,6 +3665,7 @@ and fmt_label_declaration c ctx ?(last = false) decl = ( hovbox 2 ( fmt_mutable_flag ~pro:noop ~epi:(fmt "@ ") c pld_mutable + $ fmt_if is_global "global_ " $ fmt_str_loc c pld_name $ fmt_if field_loose " " $ fmt ":" ) $ fmt "@ " @@ -3391,11 +3710,16 @@ and fmt_constructor_declaration c ctx ~first ~last:_ cstr_decl = $ fmt_attributes_and_docstrings c pcd_attributes ) $ Cmts.fmt_after c pcd_loc ) +and fmt_core_type_gf c ctx typ = + let {ptyp_attributes; _} = typ in + let is_global, _ = split_global_flags_from_attrs c.conf ptyp_attributes in + fmt_if is_global "global_ " $ fmt_core_type c (sub_typ ~ctx typ) + and fmt_constructor_arguments ?vars c ctx ~pre = function | Pcstr_tuple [] -> noop | Pcstr_tuple typs -> pre $ fmt "@ " $ fmt_opt vars - $ hvbox 0 (list typs "@ * " (sub_typ ~ctx >> fmt_core_type c)) + $ hvbox 0 (list typs "@ * " (fmt_core_type_gf c ctx)) | Pcstr_record (loc, lds) -> let p = Params.get_record_type c.conf in let fmt_ld ~first ~last x = @@ -3626,6 +3950,27 @@ and fmt_module_type c ?(rec_ = false) ({ast= mty; _} as xmty) = { empty with bdy= fmt_longident_loc c lid ; epi= Some (fmt_attributes c pmty_attributes ~pre:(Break (1, 0))) } + | Pmty_strengthen (mty, lid) -> + let {pro; psp; bdy; esp; epi; opn= _; cls= _} = + fmt_module_type c (sub_mty ~ctx mty) + in + let epi1 = + Cmts.fmt_after c pmty_loc + $ fmt_attributes c pmty_attributes ~pre:(Break (1, 0)) + $ fmt_if parens ")" + in + { empty with + pro= + Option.map pro ~f:(fun pro -> + open_hvbox 0 $ fmt_if parens "(" $ pro ) + ; psp + ; bdy= + fmt_if_k (Option.is_none pro) (open_hvbox 2 $ fmt_if parens "(") + $ hvbox 0 bdy + $ fmt_if_k (Option.is_some epi) esp + $ fmt_opt epi $ str " with " $ fmt_longident_loc c lid $ close_box + ; esp= fmt_if_k (Option.is_none epi) esp + ; epi= Some epi1 } and fmt_signature c ctx itms = let update_config c i = @@ -3665,6 +4010,9 @@ and fmt_signature_item c ?ext {ast= si; _} = $ fmt_item_attributes c ~pre:(Break (1, 0)) atrs $ doc_after ) | Psig_include {pincl_mod; pincl_attributes; pincl_loc} -> + let pincl_attributes, isfunctor = + check_include_functor_attr pincl_attributes + in update_config_maybe_disabled c pincl_loc pincl_attributes @@ fun c -> let doc_before, doc_after, atrs = @@ -3672,7 +4020,10 @@ and fmt_signature_item c ?ext {ast= si; _} = fmt_docstring_around_item c ~force_before ~fit:true pincl_attributes in let keyword, ({pro; psp; bdy; esp; epi; _} as blk) = - let kwd = str "include" $ fmt_extension_suffix c ext in + let incl = + if isfunctor then fmt "include@ functor" else str "include" + in + let kwd = incl $ fmt_extension_suffix c ext in match pincl_mod with | {pmty_desc= Pmty_typeof me; pmty_loc; pmty_attributes= _} -> ( kwd @@ -3745,7 +4096,7 @@ and fmt_class_exprs ?ext c ctx cls = let xargs, xbody = match cl.pci_expr.pcl_attributes with | [] -> - Sugar.cl_fun c.cmts ~will_keep_first_ast_node:false + Sugar.cl_fun c.conf c.cmts ~will_keep_first_ast_node:false (sub_cl ~ctx cl.pci_expr) | _ -> ([], sub_cl ~ctx cl.pci_expr) in @@ -4172,7 +4523,7 @@ and fmt_module_expr ?(dock_struct = true) c ({ast= m; _} as xmod) = ( hvbox 2 (wrap_fits_breaks ~space:false c.conf "(" ")" ( str "val " - $ fmt_expression c (sub_exp ~ctx e) + $ fmt_expression c (sub_exp c.conf ~ctx e) $ opt ty1 (package_type ": ") $ opt ty2 (package_type ":> ") ) ) $ fmt_attributes_and_docstrings c pmod_attributes ) } @@ -4242,15 +4593,19 @@ and fmt_structure_item c ~last:last_item ?ext ~semisemi | Pstr_eval (exp, atrs) -> let doc, atrs = doc_atrs atrs in fmt_docstring c doc - $ cbox 0 ~name:"eval" (fmt_expression c (sub_exp ~ctx exp)) + $ cbox 0 ~name:"eval" (fmt_expression c (sub_exp c.conf ~ctx exp)) $ fmt_item_attributes c ~pre:Space atrs | Pstr_exception extn_constr -> let pre = str "exception" $ fmt_extension_suffix c ext $ fmt "@ " in hvbox 2 ~name:"exn" (fmt_type_exception ~pre c ctx extn_constr) | Pstr_include {pincl_mod; pincl_attributes= attributes; pincl_loc} -> + let attributes, isfunctor = check_include_functor_attr attributes in update_config_maybe_disabled c pincl_loc attributes @@ fun c -> - let keyword = str "include" $ fmt_extension_suffix c ext $ fmt "@ " in + let incl = + if isfunctor then fmt "include@ functor" else str "include" + in + let keyword = incl $ fmt_extension_suffix c ext $ fmt "@ " in fmt_module_statement c ~attributes ~keyword (sub_mod ~ctx pincl_mod) | Pstr_module mb -> fmt_module_binding c ~rec_flag:false ~first:true (sub_mb ~ctx mb) @@ -4280,7 +4635,9 @@ and fmt_structure_item c ~last:last_item ?ext ~semisemi let fmt_item c ctx ~prev ~next b = let first = Option.is_none prev in let last = Option.is_none next in - let b = Sugar.Let_binding.of_let_binding c.cmts ~ctx ~first b in + let b = + Sugar.Let_binding.of_let_binding c.conf c.cmts ~ctx ~first b + in let epi = match c.conf.fmt_opts.let_binding_spacing.v with | `Compact -> None @@ -4343,7 +4700,15 @@ and fmt_let c ~ext ~rec_flag ~bindings ~parens ~fmt_atrs ~fmt_expr ~body_loc $ fmt_atrs and fmt_value_binding c ~rec_flag ?ext ?in_ ?epi - {lb_op; lb_pat; lb_args; lb_typ; lb_exp; lb_attrs; lb_loc; lb_pun} = + { lb_op + ; lb_pat + ; lb_args + ; lb_typ + ; lb_exp + ; lb_attrs + ; lb_local + ; lb_loc + ; lb_pun } = update_config_maybe_disabled c lb_loc lb_attrs @@ fun c -> let lb_pun = @@ -4426,6 +4791,7 @@ and fmt_value_binding c ~rec_flag ?ext ?in_ ?epi $ fmt_extension_suffix c ext $ fmt_attributes c at_attrs $ fmt_if rec_flag " rec" + $ fmt_if lb_local " local_" $ fmt_or pat_has_cmt "@ " " " $ fmt_pattern c lb_pat ) $ fmt_if_k @@ -4588,7 +4954,7 @@ let fmt_file (type a) ~ctx ~fmt_code ~debug (fragment : a Extended_ast.t) compose_module ~f:Fn.id (fmt_module_type c (sub_mty ~ctx:(Mty mty) mty)) | Expression, e -> - fmt_expression c (sub_exp ~ctx:(Str (Ast_helper.Str.eval e)) e) + fmt_expression c (sub_exp c.conf ~ctx:(Str (Ast_helper.Str.eval e)) e) | Repl_file, l -> fmt_repl_file c ctx l | Documentation, d -> (* TODO: [source] and [cmts] should have never been computed when diff --git a/lib/Migrate_ast.mli b/lib/Migrate_ast.mli index 14049f7de2..d531c5b74c 100644 --- a/lib/Migrate_ast.mli +++ b/lib/Migrate_ast.mli @@ -32,7 +32,9 @@ module Position : sig end module Location : sig - include module type of Location + include module type of struct + include Location + end type comparator_witness @@ -80,7 +82,9 @@ module Location : sig end module Longident : sig - include module type of Longident + include module type of struct + include Longident + end val lident : string -> t (** Make a Lident from a dotless string *) diff --git a/lib/Normalize_extended_ast.ml b/lib/Normalize_extended_ast.ml index c7634794de..76c6067d87 100644 --- a/lib/Normalize_extended_ast.ml +++ b/lib/Normalize_extended_ast.ml @@ -65,6 +65,24 @@ let normalize_code conf (m : Ast_mapper.mapper) txt = let docstring (c : Conf.t) = Docstring.normalize ~parse_docstrings:c.fmt_opts.parse_docstrings.v +let normalize_jane_street_local_annotations c : attributes -> attributes = + List.map ~f:(fun attr -> + match attr with + | {attr_name= {txt= old_name; _}; attr_payload= PStr []; _} -> + let new_name txt = + {attr with attr_name= {attr.attr_name with txt}} + in + if Conf.is_jane_street_local_annotation c "local" ~test:old_name + then new_name "extension.local" + else if + Conf.is_jane_street_local_annotation c "global" ~test:old_name + then new_name "extension.global" + else if + Conf.is_jane_street_local_annotation c "exclave" ~test:old_name + then new_name "extension.exclave" + else attr + | _ -> attr ) + let sort_attributes : attributes -> attributes = List.sort ~compare:Poly.compare @@ -140,19 +158,71 @@ let make_mapper conf ~ignore_doc_comments = (Exp.sequence ~loc:loc1 ~attrs:attrs1 (Exp.sequence ~loc:loc2 ~attrs:attrs2 exp1 exp2) exp3 ) + | Pexp_extension (({txt= old_name; _} as old_loc_name), PStr []) + when conf.opr_opts.rewrite_old_style_jane_street_local_annotations.v -> + let new_name txt = + Exp.extension ~loc:loc1 ~attrs:attrs1 + ({old_loc_name with txt}, PStr []) + in + let exp' = + if Conf.is_jane_street_local_annotation conf "local" ~test:old_name + then new_name "extension.local" + else if + Conf.is_jane_street_local_annotation conf "exclave" + ~test:old_name + then new_name "extension.exclave" + else exp + in + Ast_mapper.default_mapper.expr m exp' | _ -> Ast_mapper.default_mapper.expr m exp in + (* The old-style [[@local]] attributes can only occur in record label + declarations, types, and patterns; checking there explicitly ensures we + don't confuse them with the existing [let[@local always] f x = x] + attribute, which occurs at a different level. *) let typ (m : Ast_mapper.mapper) typ = + (* Types also need their location stack cleared *) let typ = {typ with ptyp_loc_stack= []} in + let typ = + if conf.opr_opts.rewrite_old_style_jane_street_local_annotations.v then + { typ with + ptyp_attributes= + normalize_jane_street_local_annotations conf typ.ptyp_attributes + } + else typ + in Ast_mapper.default_mapper.typ m typ in + let pat (m : Ast_mapper.mapper) pat = + let pat = + if conf.opr_opts.rewrite_old_style_jane_street_local_annotations.v then + { pat with + ppat_attributes= + normalize_jane_street_local_annotations conf pat.ppat_attributes + } + else pat + in + Ast_mapper.default_mapper.pat m pat + in + let label_declaration (m : Ast_mapper.mapper) ld = + let ld = + if conf.opr_opts.rewrite_old_style_jane_street_local_annotations.v then + { ld with + pld_attributes= + normalize_jane_street_local_annotations conf ld.pld_attributes } + else ld + in + Ast_mapper.default_mapper.label_declaration m ld + in { Ast_mapper.default_mapper with location ; attribute ; attributes ; repl_phrase ; expr - ; typ } + ; pat + ; typ + ; label_declaration } let ast fragment ~ignore_doc_comments c = map fragment (make_mapper c ~ignore_doc_comments) diff --git a/lib/Normalize_std_ast.ml b/lib/Normalize_std_ast.ml index a037bc8a89..11c3d27314 100644 --- a/lib/Normalize_std_ast.ml +++ b/lib/Normalize_std_ast.ml @@ -16,6 +16,9 @@ let is_doc = function | {attr_name= {Location.txt= "ocaml.doc" | "ocaml.text"; _}; _} -> true | _ -> false +let is_erasable_jane_syntax attr = + String.is_prefix ~prefix:"jane.erasable." attr.attr_name.txt + let dedup_cmts fragment ast comments = let of_ast ast = let docs = ref (Set.empty (module Cmt)) in @@ -66,7 +69,7 @@ let docstring (c : Conf.t) = let sort_attributes : attributes -> attributes = List.sort ~compare:Poly.compare -let make_mapper conf ~ignore_doc_comments = +let make_mapper conf ~ignore_doc_comments ~erase_jane_syntax = let open Ast_helper in (* remove locations *) let location _ _ = Location.none in @@ -100,6 +103,11 @@ let make_mapper conf ~ignore_doc_comments = in (* sort attributes *) let attributes (m : Ast_mapper.mapper) (atrs : attribute list) = + let atrs = + if erase_jane_syntax then + List.filter atrs ~f:(fun a -> not (is_erasable_jane_syntax a)) + else atrs + in let atrs = if ignore_doc_comments then List.filter atrs ~f:(fun a -> not (is_doc a)) @@ -200,12 +208,15 @@ let make_mapper conf ~ignore_doc_comments = ; pat ; typ } -let ast fragment ~ignore_doc_comments c = - map fragment (make_mapper c ~ignore_doc_comments) +let ast fragment ~ignore_doc_comments ~erase_jane_syntax c = + map fragment (make_mapper c ~ignore_doc_comments ~erase_jane_syntax) -let equal fragment ~ignore_doc_comments c ast1 ast2 = +let equal fragment ~ignore_doc_comments ~erase_jane_syntax c ~old:ast1 + ~new_:ast2 = let map = ast fragment c ~ignore_doc_comments in - equal fragment (map ast1) (map ast2) + equal fragment + (map ~erase_jane_syntax ast1) + (map ~erase_jane_syntax:false ast2) let ast = ast ~ignore_doc_comments:false @@ -236,15 +247,21 @@ let docstrings (type a) (fragment : a t) s = let (_ : a) = map fragment (make_docstring_mapper docstrings) s in !docstrings -let docstring conf = - let mapper = make_mapper conf ~ignore_doc_comments:false in +let docstring conf ~erase_jane_syntax = + let mapper = + make_mapper conf ~ignore_doc_comments:false ~erase_jane_syntax + in let normalize_code = normalize_code conf mapper in docstring conf ~normalize_code -let moved_docstrings fragment c s1 s2 = +let moved_docstrings fragment ~erase_jane_syntax c ~old:s1 ~new_:s2 = let d1 = docstrings fragment s1 in let d2 = docstrings fragment s2 in - let equal (_, x) (_, y) = String.equal (docstring c x) (docstring c y) in + let equal ~old:(_, x) ~new_:(_, y) = + String.equal + (docstring c x ~erase_jane_syntax) + (docstring c y ~erase_jane_syntax:false) + in let cmt_kind = `Doc_comment in let cmt (loc, x) = Cmt.create_docstring x loc in let dropped x = {Cmt.kind= `Dropped (cmt x); cmt_kind} in @@ -253,11 +270,17 @@ let moved_docstrings fragment c s1 s2 = match List.zip d1 d2 with | Unequal_lengths -> (* We only return the ones that are not in both lists. *) - let l1 = List.filter d1 ~f:(fun x -> not (List.mem ~equal d2 x)) in + let l1 = + List.filter d1 ~f:(fun old -> + List.for_all d2 ~f:(fun new_ -> not (equal ~old ~new_)) ) + in let l1 = List.map ~f:dropped l1 in - let l2 = List.filter d2 ~f:(fun x -> not (List.mem ~equal d1 x)) in + let l2 = + List.filter d2 ~f:(fun new_ -> + List.for_all d1 ~f:(fun old -> not (equal ~old ~new_)) ) + in let l2 = List.map ~f:added l2 in List.rev_append l1 l2 | Ok l -> - let l = List.filter l ~f:(fun (x, y) -> not (equal x y)) in + let l = List.filter l ~f:(fun (old, new_) -> not (equal ~old ~new_)) in List.map ~f:modified l diff --git a/lib/Normalize_std_ast.mli b/lib/Normalize_std_ast.mli index 880c2ea3f7..5245daf2d6 100644 --- a/lib/Normalize_std_ast.mli +++ b/lib/Normalize_std_ast.mli @@ -9,11 +9,27 @@ (* *) (**************************************************************************) -val ast : 'a Std_ast.t -> Conf.t -> 'a -> 'a -(** Normalize an AST fragment. *) +val ast : 'a Std_ast.t -> erase_jane_syntax:bool -> Conf.t -> 'a -> 'a +(** Normalize an AST fragment. If [erase_jane_syntax] is true, remove all + [Jane_syntax] attributes signaling erasable syntax. *) val equal : - 'a Std_ast.t -> ignore_doc_comments:bool -> Conf.t -> 'a -> 'a -> bool -(** Compare fragments for equality up to normalization. *) + 'a Std_ast.t + -> ignore_doc_comments:bool + -> erase_jane_syntax:bool + -> Conf.t + -> old:'a + -> new_:'a + -> bool +(** Compare fragments for equality up to normalization. If + [erase_jane_syntax] is true, first removes all [Jane_syntax] attributes + signaling erasable syntax from the [old] AST fragment; the [new_] AST + fragment should already omit them. *) -val moved_docstrings : 'a Std_ast.t -> Conf.t -> 'a -> 'a -> Cmt.error list +val moved_docstrings : + 'a Std_ast.t + -> erase_jane_syntax:bool + -> Conf.t + -> old:'a + -> new_:'a + -> Cmt.error list diff --git a/lib/Params.ml b/lib/Params.ml index 080282b903..6229a62613 100644 --- a/lib/Params.ml +++ b/lib/Params.ml @@ -208,7 +208,7 @@ let get_cases (c : Conf.t) ~ctx ~first ~last ~xbch:({ast; _} as xast) = let parens_branch, expr_parens = if align_nested_match then (false, Some false) else if c.fmt_opts.leading_nested_match_parens.v then (false, None) - else (parenze_exp xast && not body_has_parens, Some false) + else (parenze_exp c xast && not body_has_parens, Some false) in let indent = if align_nested_match then 0 else indent in let open_paren_branch, close_paren_branch, branch_expr = @@ -220,7 +220,7 @@ let get_cases (c : Conf.t) ~ctx ~first ~last ~xbch:({ast; _} as xast) = in fits_breaks " end" ~level:1 ~hint:(1000, offset) "end" in - (fmt "@;<1 0>begin", close_paren, sub_exp ~ctx:(Exp ast) nested_exp) + (fmt "@;<1 0>begin", close_paren, sub_exp c ~ctx:(Exp ast) nested_exp) | _ -> let close_paren = fmt_if_k parens_branch @@ -413,6 +413,20 @@ let get_list_expr (c : Conf.t) = let get_array_expr (c : Conf.t) = collection_expr c ~space_around:c.fmt_opts.space_around_arrays.v "[|" "|]" +let get_iarray_expr (c : Conf.t) = + collection_expr c ~space_around:c.fmt_opts.space_around_arrays.v "[:" ":]" + +(* Modeled after [collection_expr] in [`After] mode *) +let wrap_comprehension (c : Conf.t) ~space_around ~punctuation comp = + let opn = "[" ^ punctuation in + let cls = punctuation ^ "]" in + let space = if space_around then 1 else 0 in + if c.fmt_opts.dock_collection_brackets.v then + hvbox 0 + (wrap_k (str opn) (str cls) + (break space 2 $ hvbox 0 comp $ break space 0) ) + else hvbox 0 (wrap_collec c ~space_around opn cls comp) + let box_pattern_docked (c : Conf.t) ~ctx ~space_around opn cls k = let space = if space_around then 1 else 0 in let indent_opn, indent_cls = @@ -453,6 +467,10 @@ let get_array_pat (c : Conf.t) ~ctx = collection_pat c ~ctx ~space_around:c.fmt_opts.space_around_arrays.v "[|" "|]" +let get_iarray_pat (c : Conf.t) ~ctx = + collection_pat c ~ctx ~space_around:c.fmt_opts.space_around_arrays.v "[:" + ":]" + type if_then_else = { box_branch: Fmt.t -> Fmt.t ; cond: Fmt.t @@ -473,7 +491,7 @@ let get_if_then_else (c : Conf.t) ~first ~last ~parens_bch ~parens_prev_bch let ast = xbch.Ast.ast in match ast with | {pexp_desc= Pexp_beginend nested_exp; pexp_attributes= []; _} -> - (true, sub_exp ~ctx:(Exp ast) nested_exp) + (true, sub_exp c ~ctx:(Exp ast) nested_exp) | _ -> (false, xbch) in let wrap_parens ~wrap_breaks k = diff --git a/lib/Params.mli b/lib/Params.mli index 7b1db72c95..4192e29a28 100644 --- a/lib/Params.mli +++ b/lib/Params.mli @@ -123,6 +123,11 @@ val get_list_expr : Conf.t -> elements_collection val get_array_expr : Conf.t -> elements_collection +val get_iarray_expr : Conf.t -> elements_collection + +val wrap_comprehension : + Conf.t -> space_around:bool -> punctuation:string -> Fmt.t -> Fmt.t + val get_record_pat : Conf.t -> ctx:Ast.t -> elements_collection * elements_collection_record_pat @@ -130,6 +135,8 @@ val get_list_pat : Conf.t -> ctx:Ast.t -> elements_collection val get_array_pat : Conf.t -> ctx:Ast.t -> elements_collection +val get_iarray_pat : Conf.t -> ctx:Ast.t -> elements_collection + type if_then_else = { box_branch: Fmt.t -> Fmt.t ; cond: Fmt.t diff --git a/lib/Std_ast.ml b/lib/Std_ast.ml index f1c04ab441..e4119220b9 100644 --- a/lib/Std_ast.ml +++ b/lib/Std_ast.ml @@ -12,6 +12,9 @@ open Parser_standard include Parsetree +(* we always want all extensions enabled in ocamlformat *) +let () = Language_extension.enable_maximal () + type use_file = toplevel_phrase list type 'a t = diff --git a/lib/Sugar.ml b/lib/Sugar.ml index fb3c8f2b3f..387fa567ed 100644 --- a/lib/Sugar.ml +++ b/lib/Sugar.ml @@ -14,11 +14,52 @@ open Asttypes open Ast open Extended_ast +let check_local_attr conf attrs = + match + List.partition_tf attrs ~f:(fun attr -> + Conf.is_jane_street_local_annotation conf "local" + ~test:attr.attr_name.txt ) + with + | [], _ -> (attrs, false) + | _ :: _, rest -> (rest, true) + +(* This function pulls apart an arrow type, pulling out local attributes into + bools and producing a context without those attributes. This addresses the + problem that we need to remove the local attributes so that they can be + printed specially, and that the context needs to be updated to reflect + this to pass some internal ocamlformat sanity checks. It's not the + cleanest solution in a vacuum, but is perhaps the one that will cause the + fewest merge conflicts in the future. *) +let decompose_arrow conf ctx ctl ct2 = + let pull_out_local ap = + let ptyp_attributes, local = + check_local_attr conf ap.pap_type.ptyp_attributes + in + ({ap with pap_type= {ap.pap_type with ptyp_attributes}}, local) + in + let args = List.map ~f:pull_out_local ctl in + let ((res_ap, _) as res) = + let ptyp_attributes, local = check_local_attr conf ct2.ptyp_attributes in + let ap = + { pap_label= Nolabel + ; pap_loc= ct2.ptyp_loc + ; pap_type= {ct2 with ptyp_attributes} } + in + (ap, local) + in + let ctx_typ = Ptyp_arrow (List.map ~f:fst args, res_ap.pap_type) in + let ctx = + match ctx with + | Typ cty -> Typ {cty with ptyp_desc= ctx_typ} + | _ -> assert false + in + (args, res, ctx) + type arg_kind = - | Val of arg_label * pattern xt * expression xt option + | Val of bool * arg_label * pattern xt * expression xt option | Newtypes of string loc list -let fun_ cmts ?(will_keep_first_ast_node = true) xexp = +let fun_ conf cmts ?(will_keep_first_ast_node = true) xexp = let rec fun_ ?(will_keep_first_ast_node = false) ({ast= exp; _} as xexp) = let ctx = Exp exp in let {pexp_desc; pexp_loc; pexp_attributes; _} = exp in @@ -28,18 +69,28 @@ let fun_ cmts ?(will_keep_first_ast_node = true) xexp = if not will_keep_first_ast_node then Cmts.relocate cmts ~src:pexp_loc ~before:pattern.ppat_loc ~after:body.pexp_loc ; - let xargs, xbody = fun_ (sub_exp ~ctx body) in + let xargs, xbody = fun_ (sub_exp conf ~ctx body) in + let islocal, pat = + match check_local_attr conf pattern.ppat_attributes with + | _, false -> (false, sub_pat ~ctx pattern) + | ppat_attributes, true -> + let pattern = {pattern with ppat_attributes} in + let ctx = + Exp + { exp with + pexp_desc= Pexp_fun (label, default, pattern, body) } + in + (true, sub_pat ~ctx pattern) + in ( Val - ( label - , sub_pat ~ctx pattern - , Option.map default ~f:(sub_exp ~ctx) ) + (islocal, label, pat, Option.map default ~f:(sub_exp conf ~ctx)) :: xargs , xbody ) | Pexp_newtype (name, body) -> if not will_keep_first_ast_node then Cmts.relocate cmts ~src:pexp_loc ~before:body.pexp_loc ~after:body.pexp_loc ; - let xargs, xbody = fun_ (sub_exp ~ctx body) in + let xargs, xbody = fun_ (sub_exp conf ~ctx body) in let xargs = match xargs with | Newtypes names :: xargs -> Newtypes (name :: names) :: xargs @@ -51,7 +102,7 @@ let fun_ cmts ?(will_keep_first_ast_node = true) xexp = in fun_ ~will_keep_first_ast_node xexp -let cl_fun ?(will_keep_first_ast_node = true) cmts xexp = +let cl_fun ?(will_keep_first_ast_node = true) conf cmts xexp = let rec fun_ ?(will_keep_first_ast_node = false) ({ast= exp; _} as xexp) = let ctx = Cl exp in let {pcl_desc; pcl_loc; pcl_attributes; _} = exp in @@ -62,10 +113,20 @@ let cl_fun ?(will_keep_first_ast_node = true) cmts xexp = Cmts.relocate cmts ~src:pcl_loc ~before:pattern.ppat_loc ~after:body.pcl_loc ; let xargs, xbody = fun_ (sub_cl ~ctx body) in + let islocal, pat = + match check_local_attr conf pattern.ppat_attributes with + | _, false -> (false, sub_pat ~ctx pattern) + | ppat_attributes, true -> + let pattern = {pattern with ppat_attributes} in + let ctx = + Cl + { exp with + pcl_desc= Pcl_fun (label, default, pattern, body) } + in + (true, sub_pat ~ctx pattern) + in ( Val - ( label - , sub_pat ~ctx pattern - , Option.map default ~f:(sub_exp ~ctx) ) + (islocal, label, pat, Option.map default ~f:(sub_exp conf ~ctx)) :: xargs , xbody ) | _ -> ([], xexp) @@ -74,7 +135,7 @@ let cl_fun ?(will_keep_first_ast_node = true) cmts xexp = fun_ ~will_keep_first_ast_node xexp module Exp = struct - let infix cmts prec xexp = + let infix conf cmts prec xexp = let assoc = Option.value_map prec ~default:Assoc.Non ~f:Assoc.of_prec in let rec infix_ ?(child_expr = true) xop xexp = let ctx = Exp xexp.ast in @@ -87,7 +148,7 @@ module Exp = struct , {pexp_desc= Pexp_infix ({txt= op; loc}, e1, e2); pexp_loc= src; _} ) when Option.equal Prec.equal prec (prec_ast ctx) -> - let op_args1 = infix_ None (sub_exp ~ctx e1) in + let op_args1 = infix_ None (sub_exp conf ~ctx e1) in let before = match op_args1 with | (Some {loc; _}, _) :: _ -> loc @@ -96,12 +157,14 @@ module Exp = struct in if child_expr then Cmts.relocate cmts ~src ~before ~after:e2.pexp_loc ; - op_args1 @ [(Some {txt= op; loc}, sub_exp ~ctx e2)] + op_args1 @ [(Some {txt= op; loc}, sub_exp conf ~ctx e2)] | ( Right , {pexp_desc= Pexp_infix ({txt= op; loc}, e1, e2); pexp_loc= src; _} ) when Option.equal Prec.equal prec (prec_ast ctx) -> - let op_args2 = infix_ (Some {txt= op; loc}) (sub_exp ~ctx e2) in + let op_args2 = + infix_ (Some {txt= op; loc}) (sub_exp conf ~ctx e2) + in let before = match xop with Some op -> op.loc | None -> e1.pexp_loc in @@ -111,13 +174,13 @@ module Exp = struct | None -> e1.pexp_loc in if child_expr then Cmts.relocate cmts ~src ~before ~after ; - (xop, sub_exp ~ctx e1) :: op_args2 + (xop, sub_exp conf ~ctx e1) :: op_args2 | _ -> [(xop, xexp)] in infix_ None ~child_expr:false xexp end -let sequence cmts xexp = +let sequence conf cmts xexp = let rec sequence_ ?(allow_attribute = true) ({ast= exp; _} as xexp) = let ctx = Exp exp in let {pexp_desc; pexp_loc; _} = exp in @@ -142,12 +205,16 @@ let sequence cmts xexp = ~after:e2.pexp_loc ; Cmts.relocate cmts ~src:pexp_loc ~before:e1.pexp_loc ~after:e2.pexp_loc ; - if Ast.exposed_right_exp Ast.Let_match e1 then - [(None, sub_exp ~ctx e1); (Some ext, sub_exp ~ctx e2)] + if Ast.exposed_right_exp conf Ast.Let_match e1 then + [(None, sub_exp conf ~ctx e1); (Some ext, sub_exp conf ~ctx e2)] else - let l1 = sequence_ ~allow_attribute:false (sub_exp ~ctx e1) in + let l1 = + sequence_ ~allow_attribute:false (sub_exp conf ~ctx e1) + in let l2 = - match sequence_ ~allow_attribute:false (sub_exp ~ctx e2) with + match + sequence_ ~allow_attribute:false (sub_exp conf ~ctx e2) + with | [] -> [] | (_, e2) :: l2 -> (Some ext, e2) :: l2 in @@ -158,12 +225,12 @@ let sequence cmts xexp = else ( Cmts.relocate cmts ~src:pexp_loc ~before:e1.pexp_loc ~after:e2.pexp_loc ; - if Ast.exposed_right_exp Ast.Let_match e1 then - [(None, sub_exp ~ctx e1); (None, sub_exp ~ctx e2)] + if Ast.exposed_right_exp conf Ast.Let_match e1 then + [(None, sub_exp conf ~ctx e1); (None, sub_exp conf ~ctx e2)] else List.append - (sequence_ ~allow_attribute:false (sub_exp ~ctx e1)) - (sequence_ ~allow_attribute:false (sub_exp ~ctx e2)) ) + (sequence_ ~allow_attribute:false (sub_exp conf ~ctx e1)) + (sequence_ ~allow_attribute:false (sub_exp conf ~ctx e2)) ) | _ -> [(None, xexp)] in sequence_ xexp @@ -180,16 +247,16 @@ let mod_with pmty = let l_rev, m = mod_with_ pmty in (List.rev l_rev, m) -let rec polynewtype_ cmts pvars body relocs = +let rec polynewtype_ conf cmts pvars body relocs = let ctx = Exp body in match (pvars, body.pexp_desc) with | [], Pexp_constraint (exp, typ) -> let relocs = (body.pexp_loc, exp.pexp_loc) :: relocs in - Some (sub_typ ~ctx typ, sub_exp ~ctx exp, relocs) + Some (sub_typ ~ctx typ, sub_exp conf ~ctx exp, relocs) | pvar :: pvars, Pexp_newtype (nvar, exp) when String.equal pvar.txt nvar.txt -> let relocs = (nvar.loc, pvar.loc) :: relocs in - polynewtype_ cmts pvars exp relocs + polynewtype_ conf cmts pvars exp relocs | _ -> None (** [polynewtype cmts pat exp] returns expression of a type-constrained @@ -204,11 +271,13 @@ let rec polynewtype_ cmts pvars body relocs = {[ let f : type r s. r s t = e ]} *) -let polynewtype cmts pat body = +let polynewtype conf cmts pat body = let ctx = Pat pat in match pat.ppat_desc with | Ppat_constraint (pat2, {ptyp_desc= Ptyp_poly (pvars, _); _}) -> ( - match polynewtype_ cmts pvars body [(pat.ppat_loc, pat2.ppat_loc)] with + match + polynewtype_ conf cmts pvars body [(pat.ppat_loc, pat2.ppat_loc)] + with | Some (typ, exp, relocs) -> List.iter relocs ~f:(fun (src, dst) -> Cmts.relocate cmts ~src ~before:dst ~after:dst ) ; @@ -229,9 +298,10 @@ module Let_binding = struct ; lb_exp: expression xt ; lb_pun: bool ; lb_attrs: attribute list + ; lb_local: bool ; lb_loc: Location.t } - let split_annot cmts xargs ({ast= body; _} as xbody) = + let split_annot conf cmts xargs ({ast= body; _} as xbody) = let ctx = Exp body in match body.pexp_desc with | Pexp_constraint (exp, typ) @@ -247,35 +317,141 @@ module Let_binding = struct let pat = Ast_helper.Pat.any () in Exp (Ast_helper.Exp.fun_ Nolabel None pat exp) in - (xargs, `Other (sub_typ ~ctx:typ_ctx typ), sub_exp ~ctx:exp_ctx exp) + ( xargs + , `Other (sub_typ ~ctx:typ_ctx typ) + , sub_exp conf ~ctx:exp_ctx exp ) (* The type constraint is always printed before the declaration for functions, for other value bindings we preserve its position. *) | Pexp_constraint (exp, typ) when not (List.is_empty xargs) -> Cmts.relocate cmts ~src:body.pexp_loc ~before:exp.pexp_loc ~after:exp.pexp_loc ; - (xargs, `Other (sub_typ ~ctx typ), sub_exp ~ctx exp) + (xargs, `Other (sub_typ ~ctx typ), sub_exp conf ~ctx exp) | Pexp_coerce (exp, typ1, typ2) when Source.type_constraint_is_first typ2 exp.pexp_loc -> Cmts.relocate cmts ~src:body.pexp_loc ~before:exp.pexp_loc ~after:exp.pexp_loc ; let typ1 = Option.map typ1 ~f:(sub_typ ~ctx) in - (xargs, `Coerce (typ1, sub_typ ~ctx typ2), sub_exp ~ctx exp) + (xargs, `Coerce (typ1, sub_typ ~ctx typ2), sub_exp conf ~ctx exp) | _ -> (xargs, `None, xbody) - let split_fun_args cmts xpat xbody = + let split_fun_args conf cmts xpat xbody = let xargs, xbody = match xpat.ast with | {ppat_desc= Ppat_var _; ppat_attributes= []; _} -> - fun_ cmts ~will_keep_first_ast_node:false xbody + fun_ conf cmts ~will_keep_first_ast_node:false xbody | _ -> ([], xbody) in match (xbody.ast.pexp_desc, xpat.ast.ppat_desc) with | Pexp_constraint _, Ppat_constraint _ -> (xargs, `None, xbody) - | _ -> split_annot cmts xargs xbody + | _ -> split_annot conf cmts xargs xbody + + (** Conservatively decides when to use the [let local_ ...] sugar. + + Putting [local_] on the left-hand-side of a simple variable binding + is sugar for putting it on the right-hand-side. + + {[ + let x = local_ expression (* parses the exact same as *) + let local_ x = expression + ]} - let type_cstr cmts ~ctx lb_pat lb_exp = + We have to be careful about trying to sugar something that isn't + already sugared, though. + + {[ + let (x, y) = local_ expression (* parses while *) + let local_ (x, y) = expression (* does not *) + + let local_ f x = expression (* parses the same as *) + let f = local_ fun x -> expression (* and not as *) + let f x = local_ expression + ]} + + Ocamlformat checks that the formatting output does not change + the parsed ast, which catches and fails on these cases. + It also, however, fails even on changes to the ast that don't + have any semantic difference. + + {[ + let x : string = local_ expression (* means the same thing as *) + let local_ x : string = expression (* but parses differently *) + ]} + + Currently, if there is any type annotation or coercion on the let + binding, then sugaring the [local_] will create a different + parsetree, so we just avoid sugaring in these cases. + *) + let local_pattern_can_be_sugared conf pvb_pat pvb_constraint exp_loc cmts = + (* If the original code was sugared, preserve that always. *) + let _, already_sugared = check_local_attr conf pvb_pat.ppat_attributes in + (* Don't wipe away comments before [local_]. *) + let comment_before = Cmts.has_before cmts exp_loc in + already_sugared + || (not comment_before) + && + match pvb_pat.ppat_desc with + | Ppat_var _ -> ( + match pvb_constraint with + | None -> + (* [ let x = local_ "hi" ] *) + true + | Some (Pvc_constraint _) -> + (* [ let x : string = local_ "hi" ] [ let x : 'a. string = + local_ "hi" ] *) + false + | Some (Pvc_coercion _) -> + (* [ let x : string :> string = local_ "hi" ] [ let x :> string + = local_ "hi" ] *) + false ) + | _ -> false + + let maybe_sugar_local conf cmts ~ctx pvb_pat pvb_expr pvb_is_pun + pvb_constraint = + let is_local_pattern, ctx, pvb_pat, pvb_expr = + match pvb_expr.pexp_desc with + | Pexp_apply + ( { pexp_desc= Pexp_extension ({txt= extension_local; _}, PStr []) + ; _ } + , [(Nolabel, sbody)] ) + when Conf.is_jane_street_local_annotation conf "local" + ~test:extension_local -> + let is_local_pattern, sbody = + (* The pattern part must still be rewritten as the parser + duplicated the type annotations and extensions into the + pattern and the expression. *) + if + local_pattern_can_be_sugared conf pvb_pat pvb_constraint + pvb_expr.pexp_loc cmts + then + let sattrs, _ = check_local_attr conf sbody.pexp_attributes in + (true, {sbody with pexp_attributes= sattrs}) + else (false, pvb_expr) + in + let pattrs, _ = check_local_attr conf pvb_pat.ppat_attributes in + let pat = {pvb_pat with ppat_attributes= pattrs} in + let fake_ctx = + Lb + { pvb_pat= pat + ; pvb_expr= sbody + ; pvb_is_pun + ; pvb_attributes= [] + ; pvb_loc= Location.none + ; pvb_constraint= None } + in + (is_local_pattern, fake_ctx, pat, sbody) + | _ -> (false, ctx, pvb_pat, pvb_expr) + in + let lb_pat = sub_pat ~ctx pvb_pat + and lb_exp = sub_exp conf ~ctx pvb_expr in + (is_local_pattern, lb_pat, lb_exp) + + let type_cstr conf cmts ~ctx pvb_pat pvb_expr pvb_is_pun pvb_constraint = + let is_local_pattern, lb_pat, lb_exp = + maybe_sugar_local conf cmts ~ctx pvb_pat pvb_expr pvb_is_pun + pvb_constraint + in let ({ast= pat; _} as xpat) = - match (lb_pat.ppat_desc, lb_exp.pexp_desc) with + match (lb_pat.ast.ppat_desc, lb_exp.ast.pexp_desc) with (* recognize and undo the pattern of code introduced by ocaml/ocaml@fd0dc6a0fbf73323c37a73ea7e8ffc150059d6ff to fix https://caml.inria.fr/mantis/view.php?id=7344 *) @@ -284,35 +460,39 @@ module Let_binding = struct , {ptyp_desc= Ptyp_poly ([], typ1); _} ) , Pexp_constraint (_, typ2) ) when equal_core_type typ1 typ2 -> - Cmts.relocate cmts ~src:lb_pat.ppat_loc ~before:pat.ppat_loc + Cmts.relocate cmts ~src:lb_pat.ast.ppat_loc ~before:pat.ppat_loc ~after:pat.ppat_loc ; - sub_pat ~ctx:(Pat lb_pat) pat + sub_pat ~ctx:(Pat lb_pat.ast) pat | ( Ppat_constraint (_, {ptyp_desc= Ptyp_poly (_, typ1); _}) , Pexp_coerce (_, _, typ2) ) when equal_core_type typ1 typ2 -> - sub_pat ~ctx lb_pat - | _ -> sub_pat ~ctx lb_pat + sub_pat ~ctx lb_pat.ast + | _ -> sub_pat ~ctx lb_pat.ast in let pat_is_extension {ppat_desc; _} = match ppat_desc with Ppat_extension _ -> true | _ -> false in - let ({ast= body; _} as xbody) = sub_exp ~ctx lb_exp in - if - (not (List.is_empty xbody.ast.pexp_attributes)) || pat_is_extension pat - then (xpat, [], `None, xbody) - else - match polynewtype cmts pat body with - | Some (xpat, pvars, xtyp, xbody) -> - (xpat, [], `Polynewtype (pvars, xtyp), xbody) - | None -> - let xpat = - match xpat.ast.ppat_desc with - | Ppat_constraint (p, {ptyp_desc= Ptyp_poly ([], _); _}) -> - sub_pat ~ctx:xpat.ctx p - | _ -> xpat - in - let xargs, typ, xbody = split_fun_args cmts xpat xbody in - (xpat, xargs, typ, xbody) + let ({ast= body; _} as xbody) = sub_exp conf ~ctx lb_exp.ast in + let pat, xargs, typ, exp = + if + (not (List.is_empty xbody.ast.pexp_attributes)) + || pat_is_extension pat + then (xpat, [], `None, xbody) + else + match polynewtype conf cmts pat body with + | Some (xpat, pvars, xtyp, xbody) -> + (xpat, [], `Polynewtype (pvars, xtyp), xbody) + | None -> + let xpat = + match xpat.ast.ppat_desc with + | Ppat_constraint (p, {ptyp_desc= Ptyp_poly ([], _); _}) -> + sub_pat ~ctx:xpat.ctx p + | _ -> xpat + in + let xargs, typ, xbody = split_fun_args conf cmts xpat xbody in + (xpat, xargs, typ, xbody) + in + (is_local_pattern, pat, xargs, typ, exp) let typ_of_pvb_constraint ~ctx = function | Some (Pvc_constraint {locally_abstract_univars= []; typ}) -> @@ -328,15 +508,16 @@ module Let_binding = struct | {ppat_desc= Ppat_var _; ppat_attributes= []; _}, `None -> true | _ -> false - let of_let_binding cmts ~ctx ~first + let of_let_binding conf cmts ~ctx ~first {pvb_pat; pvb_expr; pvb_constraint; pvb_is_pun; pvb_attributes; pvb_loc} = - let lb_exp = sub_exp ~ctx pvb_expr - and lb_pat = sub_pat ~ctx pvb_pat + let islocal, lb_pat, lb_exp = + maybe_sugar_local conf cmts ~ctx pvb_pat pvb_expr pvb_is_pun + pvb_constraint and lb_typ = typ_of_pvb_constraint ~ctx pvb_constraint in let lb_args, lb_typ, lb_exp = if should_desugar_args lb_pat lb_typ then - split_fun_args cmts lb_pat lb_exp + split_fun_args conf cmts lb_pat lb_exp else ([], lb_typ, lb_exp) in { lb_op= Location.{txt= (if first then "let" else "and"); loc= none} @@ -346,15 +527,16 @@ module Let_binding = struct ; lb_exp ; lb_pun= pvb_is_pun ; lb_attrs= pvb_attributes + ; lb_local= islocal ; lb_loc= pvb_loc } - let of_let_bindings cmts ~ctx = - List.mapi ~f:(fun i -> of_let_binding cmts ~ctx ~first:(i = 0)) + let of_let_bindings conf cmts ~ctx = + List.mapi ~f:(fun i -> of_let_binding conf cmts ~ctx ~first:(i = 0)) - let of_binding_ops cmts ~ctx bos = + let of_binding_ops conf cmts ~ctx bos = List.map bos ~f:(fun bo -> - let lb_pat, lb_args, lb_typ, lb_exp = - type_cstr cmts ~ctx bo.pbop_pat bo.pbop_exp + let islocal, lb_pat, lb_args, lb_typ, lb_exp = + type_cstr conf cmts ~ctx bo.pbop_pat bo.pbop_exp false None in { lb_op= bo.pbop_op ; lb_pat @@ -367,5 +549,6 @@ module Let_binding = struct String.equal v e | _ -> false ) ; lb_attrs= [] + ; lb_local= islocal ; lb_loc= bo.pbop_loc } ) end diff --git a/lib/Sugar.mli b/lib/Sugar.mli index 43eebe2013..b43267ec72 100644 --- a/lib/Sugar.mli +++ b/lib/Sugar.mli @@ -13,43 +13,61 @@ open Migrate_ast open Asttypes open Extended_ast +val decompose_arrow : + Conf.t + -> Ast.t + -> arrow_param list + -> core_type + -> (arrow_param * bool) list * (arrow_param * bool) * Ast.t +(** [decompose_arrow ctl ct2] returns a list of arrow params, where the last + is a dummy param corresponding to ct2 (the return type) and a bool + indicating the presence of a local attribute (which has been removed). + The returned Ast.t is a ctx that has similarly been updated to remove the + attributes *) + type arg_kind = - | Val of arg_label * pattern Ast.xt * expression Ast.xt option + | Val of bool * arg_label * pattern Ast.xt * expression Ast.xt option | Newtypes of string loc list val fun_ : - Cmts.t + Conf.t + -> Cmts.t -> ?will_keep_first_ast_node:bool -> expression Ast.xt -> arg_kind list * expression Ast.xt -(** [fun_ cmts will_keep_first_ast_node exp] returns the list of arguments - and the body of the function [exp]. [will_keep_first_ast_node] is set by - default, otherwise the [exp] is returned without modification. *) +(** [fun_ conf cmts will_keep_first_ast_node exp] returns the list of + arguments and the body of the function [exp]. [will_keep_first_ast_node] + is set by default, otherwise the [exp] is returned without modification. *) val cl_fun : ?will_keep_first_ast_node:bool + -> Conf.t -> Cmts.t -> class_expr Ast.xt -> arg_kind list * class_expr Ast.xt -(** [cl_fun will_keep_first_ast_node cmts exp] returns the list of arguments - and the body of the function [exp]. [will_keep_first_ast_node] is set by - default, otherwise the [exp] is returned without modification. *) +(** [cl_fun conf will_keep_first_ast_node cmts exp] returns the list of + arguments and the body of the function [exp]. [will_keep_first_ast_node] + is set by default, otherwise the [exp] is returned without modification. *) module Exp : sig val infix : - Cmts.t + Conf.t + -> Cmts.t -> Prec.t option -> expression Ast.xt -> (string loc option * expression Ast.xt) list - (** [infix cmts prec exp] returns the infix operator and the list of + (** [infix conf cmts prec exp] returns the infix operator and the list of operands applied to this operator from expression [exp]. [prec] is the precedence of the infix operator. *) end val sequence : - Cmts.t -> expression Ast.xt -> (label loc option * expression Ast.xt) list -(** [sequence cmts exp] returns the list of expressions (with the optional - extension) from a sequence of expressions [exp]. *) + Conf.t + -> Cmts.t + -> expression Ast.xt + -> (label loc option * expression Ast.xt) list +(** [sequence conf cmts exp] returns the list of expressions (with the + optional extension) from a sequence of expressions [exp]. *) val mod_with : module_type Ast.xt @@ -71,12 +89,15 @@ module Let_binding : sig ; lb_exp: expression Ast.xt ; lb_pun: bool ; lb_attrs: attribute list + ; lb_local: bool ; lb_loc: Location.t } val of_let_binding : - Cmts.t -> ctx:Ast.t -> first:bool -> value_binding -> t + Conf.t -> Cmts.t -> ctx:Ast.t -> first:bool -> value_binding -> t - val of_let_bindings : Cmts.t -> ctx:Ast.t -> value_binding list -> t list + val of_let_bindings : + Conf.t -> Cmts.t -> ctx:Ast.t -> value_binding list -> t list - val of_binding_ops : Cmts.t -> ctx:Ast.t -> binding_op list -> t list + val of_binding_ops : + Conf.t -> Cmts.t -> ctx:Ast.t -> binding_op list -> t list end diff --git a/lib/Translation_unit.ml b/lib/Translation_unit.ml index 2013919c74..ca147b1299 100644 --- a/lib/Translation_unit.ml +++ b/lib/Translation_unit.ml @@ -323,9 +323,11 @@ let format (type a b) (fg : a Extended_ast.t) (std_fg : b Std_ast.t) | std_t_new -> Ok std_t_new in (* Ast not preserved ? *) + let erase_jane_syntax = Erase_jane_syntax.should_erase () in ( if (not - (Normalize_std_ast.equal std_fg conf std_t.ast std_t_new.ast + (Normalize_std_ast.equal std_fg conf ~old:std_t.ast + ~new_:std_t_new.ast ~erase_jane_syntax ~ignore_doc_comments:(not conf.opr_opts.comment_check.v) ) ) && not (Normalize_extended_ast.equal fg conf t.ast t_new.ast @@ -333,11 +335,12 @@ let format (type a b) (fg : a Extended_ast.t) (std_fg : b Std_ast.t) then let old_ast = dump_ast std_fg ~suffix:".old" - (Normalize_std_ast.ast std_fg conf std_t.ast) + (Normalize_std_ast.ast std_fg ~erase_jane_syntax conf std_t.ast) in let new_ast = dump_ast std_fg ~suffix:".new" - (Normalize_std_ast.ast std_fg conf std_t_new.ast) + (Normalize_std_ast.ast std_fg ~erase_jane_syntax:false conf + std_t_new.ast ) in let args ~suffix = [ ("output file", dump_formatted ~suffix fmted) @@ -347,12 +350,12 @@ let format (type a b) (fg : a Extended_ast.t) (std_fg : b Std_ast.t) Option.map f_opt ~f:(fun f -> (s, String.sexp_of_t f)) ) in if - Normalize_std_ast.equal std_fg ~ignore_doc_comments:true conf - std_t.ast std_t_new.ast + Normalize_std_ast.equal std_fg ~ignore_doc_comments:true + ~erase_jane_syntax conf ~old:std_t.ast ~new_:std_t_new.ast then let docstrings = - Normalize_std_ast.moved_docstrings std_fg conf std_t.ast - std_t_new.ast + Normalize_std_ast.moved_docstrings std_fg ~erase_jane_syntax + conf ~old:std_t.ast ~new_:std_t_new.ast in let args = args ~suffix:".unequal-docs" in internal_error @@ -363,7 +366,8 @@ let format (type a b) (fg : a Extended_ast.t) (std_fg : b Std_ast.t) internal_error [`Ast_changed] args else dump_ast std_fg ~suffix:"" - (Normalize_std_ast.ast std_fg conf std_t_new.ast) + (Normalize_std_ast.ast std_fg ~erase_jane_syntax conf + std_t_new.ast ) |> function | Some file -> if i = 1 then Format.eprintf "[DEBUG] AST structure: %s\n" file diff --git a/lib/bin_conf/Bin_conf.ml b/lib/bin_conf/Bin_conf.ml index 05280bf891..473c198e7c 100644 --- a/lib/bin_conf/Bin_conf.ml +++ b/lib/bin_conf/Bin_conf.ml @@ -30,7 +30,9 @@ type t = ; disable_conf_files: bool ; ignore_invalid_options: bool ; ocp_indent_config: bool - ; config: (string * string) list } + ; config: (string * string) list + ; erase_jane_syntax: bool + ; rewrite_old_style_jane_street_local_annotations: bool } let default = { lib_conf= Conf.default @@ -46,7 +48,9 @@ let default = ; disable_conf_files= false ; ignore_invalid_options= false ; ocp_indent_config= false - ; config= [] } + ; config= [] + ; erase_jane_syntax= false + ; rewrite_old_style_jane_street_local_annotations= false } let global_conf = ref default @@ -313,6 +317,17 @@ let ocp_indent_config = ~set:(fun ocp_indent_config conf -> {conf with ocp_indent_config}) Arg.(value & flag & info ["ocp-indent-config"] ~doc ~docs) +let erase_jane_syntax = + let doc = + "Erase all erasable Jane Street syntax extensions. Jane Street uses \ + this to generate the upstream-compatible public release code for our \ + libraries (vs. the variant with Jane Street-specific syntax). THIS \ + OPTION WILL CHANGE THE RESULTING AST." + in + declare_option + ~set:(fun erase_jane_syntax conf -> {conf with erase_jane_syntax}) + Arg.(value & flag & info ["erase-jane-syntax"] ~doc ~docs) + let terms = [ Term.( const (fun lib_conf_modif conf -> @@ -330,7 +345,8 @@ let terms = ; disable_conf_files ; ignore_invalid_options ; ocp_indent_config - ; config ] + ; config + ; erase_jane_syntax ] let global_term = let compose (t1 : ('a -> 'b) Term.t) (t2 : ('b -> 'c) Term.t) : @@ -738,6 +754,9 @@ let validate_action () = Error (Printf.sprintf "Cannot specify %s with %s" a1 a2) let validate () = + (* We have to store this globally so that we can access it in the parser, + which doesn't have a [Conf_t.t]. *) + Erase_jane_syntax.set_should_erase !global_conf.erase_jane_syntax ; let root = Option.map !global_conf.root ~f:Fpath.(fun x -> v x |> to_absolute |> normalize) diff --git a/lib/bin_conf/dune b/lib/bin_conf/dune index dc6f287f54..3d79a0f2bc 100644 --- a/lib/bin_conf/dune +++ b/lib/bin_conf/dune @@ -5,4 +5,4 @@ (:standard -open Ocaml_common -open Ocamlformat_stdlib)) (instrumentation (backend bisect_ppx)) - (libraries ocamlformat-lib re)) + (libraries erase_jane_syntax ocamlformat-lib re)) diff --git a/lib/erase_jane_syntax/dune b/lib/erase_jane_syntax/dune new file mode 100644 index 0000000000..052816cd6a --- /dev/null +++ b/lib/erase_jane_syntax/dune @@ -0,0 +1,3 @@ +(library + (public_name ocamlformat.erase_jane_syntax) + (name erase_jane_syntax)) diff --git a/lib/erase_jane_syntax/erase_jane_syntax.ml b/lib/erase_jane_syntax/erase_jane_syntax.ml new file mode 100644 index 0000000000..deb562ec79 --- /dev/null +++ b/lib/erase_jane_syntax/erase_jane_syntax.ml @@ -0,0 +1,5 @@ +let should_erase_ref = ref false + +let set_should_erase yn = should_erase_ref := yn + +let should_erase () = !should_erase_ref diff --git a/lib/erase_jane_syntax/erase_jane_syntax.mli b/lib/erase_jane_syntax/erase_jane_syntax.mli new file mode 100644 index 0000000000..ae2f172ae0 --- /dev/null +++ b/lib/erase_jane_syntax/erase_jane_syntax.mli @@ -0,0 +1,29 @@ +(** Whether or not Jane Street-specific syntax should be erased. Anything in + vendor/parser-extended that could generate one of the Jane + Street-specific constructors we add to the extended parsetree needs to + check [should_erase] and, if true, generate the corresponding "plain" + version, as per the compiler's [Jane_syntax] construction. + + When the user does request erasure, the result of ocamlformat {e will} + modify the parse tree, but will also be parseable by upstream OCaml; we + validate that the resulting parse tree is only modified to the point of + removing Jane Street-specific syntax. + + This is done as a separate library so that it can be availble from the + parser-extended Menhir parser, which we can't pass a local boolean flag + into. This library is then depended on by everything that needs to know + about this global state. While on the ocamlformat side, we could put this + in the configuration type, that would not work for the Menhir parser; + thus, we take this somewhat more invasive approach. *) + +val set_should_erase : bool -> unit +(** Toggle whether Jane Street specific parse tree components ought to be + erased from parsing/printing: [true] if they should be erased (so that + the parse tree will be modified by ocamlformat), [false] if they should + not. *) + +val should_erase : unit -> bool +(** Check whether Jane Street specific parse tree components ought to be + erased from parsing/printing: [true] if they should be erased (so that + the parse tree will be modified by ocamlformat), [false] if they should + not. *) diff --git a/test/cli/print_config.t b/test/cli/print_config.t index 278468c55f..69b0b6156d 100644 --- a/test/cli/print_config.t +++ b/test/cli/print_config.t @@ -14,6 +14,7 @@ No redundant values: quiet=false disable-conf-attrs=false version-check=true + rewrite-old-style-jane-street-local-annotations=false assignment-operator=end-line (profile conventional (file .ocamlformat:1)) break-before-in=fit-or-vertical (profile conventional (file .ocamlformat:1)) break-cases=fit (profile conventional (file .ocamlformat:1)) @@ -92,6 +93,7 @@ Redundant values from the conventional profile: quiet=false disable-conf-attrs=false version-check=true + rewrite-old-style-jane-street-local-annotations=false assignment-operator=end-line (profile conventional (file .ocamlformat:1)) break-before-in=fit-or-vertical (profile conventional (file .ocamlformat:1)) break-cases=fit (profile conventional (file .ocamlformat:1)) @@ -170,6 +172,7 @@ Redundant values from the ocamlformat profile: quiet=false disable-conf-attrs=false version-check=true + rewrite-old-style-jane-street-local-annotations=false assignment-operator=end-line (profile ocamlformat (file .ocamlformat:1)) break-before-in=fit-or-vertical (profile ocamlformat (file .ocamlformat:1)) break-cases=nested (profile ocamlformat (file .ocamlformat:1)) diff --git a/test/passing/dune.inc b/test/passing/dune.inc index 0db3020328..71c29b6852 100644 --- a/test/passing/dune.inc +++ b/test/passing/dune.inc @@ -161,6 +161,24 @@ (package ocamlformat) (action (diff tests/assignment_operator.ml.err assignment_operator.ml.stderr))) +(rule + (deps tests/.ocamlformat ) + (package ocamlformat) + (action + (with-stdout-to attr_parens.ml.stdout + (with-stderr-to attr_parens.ml.stderr + (run %{bin:ocamlformat} --margin-check %{dep:tests/attr_parens.ml}))))) + +(rule + (alias runtest) + (package ocamlformat) + (action (diff tests/attr_parens.ml attr_parens.ml.stdout))) + +(rule + (alias runtest) + (package ocamlformat) + (action (diff tests/attr_parens.ml.err attr_parens.ml.stderr))) + (rule (deps tests/.ocamlformat ) (package ocamlformat) @@ -1220,6 +1238,24 @@ (package ocamlformat) (action (diff tests/comments_in_record.ml.err comments_in_record.ml.stderr))) +(rule + (deps tests/.ocamlformat ) + (package ocamlformat) + (action + (with-stdout-to comprehensions.ml.stdout + (with-stderr-to comprehensions.ml.stderr + (run %{bin:ocamlformat} --margin-check %{dep:tests/comprehensions.ml}))))) + +(rule + (alias runtest) + (package ocamlformat) + (action (diff tests/comprehensions.ml comprehensions.ml.stdout))) + +(rule + (alias runtest) + (package ocamlformat) + (action (diff tests/comprehensions.ml.err comprehensions.ml.stderr))) + (rule (deps tests/.ocamlformat ) (package ocamlformat) @@ -2294,6 +2330,42 @@ (package ocamlformat) (action (diff tests/ifand.ml.err ifand.ml.stderr))) +(rule + (deps tests/.ocamlformat ) + (package ocamlformat) + (action + (with-stdout-to immutable_arrays.ml.stdout + (with-stderr-to immutable_arrays.ml.stderr + (run %{bin:ocamlformat} --margin-check %{dep:tests/immutable_arrays.ml}))))) + +(rule + (alias runtest) + (package ocamlformat) + (action (diff tests/immutable_arrays.ml immutable_arrays.ml.stdout))) + +(rule + (alias runtest) + (package ocamlformat) + (action (diff tests/immutable_arrays.ml.err immutable_arrays.ml.stderr))) + +(rule + (deps tests/.ocamlformat ) + (package ocamlformat) + (action + (with-stdout-to include_functor.ml.stdout + (with-stderr-to include_functor.ml.stderr + (run %{bin:ocamlformat} --margin-check %{dep:tests/include_functor.ml}))))) + +(rule + (alias runtest) + (package ocamlformat) + (action (diff tests/include_functor.ml include_functor.ml.stdout))) + +(rule + (alias runtest) + (package ocamlformat) + (action (diff tests/include_functor.ml.err include_functor.ml.stderr))) + (rule (deps tests/.ocamlformat ) (package ocamlformat) @@ -3609,6 +3681,60 @@ (package ocamlformat) (action (diff tests/loc_stack.ml.err loc_stack.ml.stderr))) +(rule + (deps tests/.ocamlformat ) + (package ocamlformat) + (action + (with-stdout-to local-erased.ml.stdout + (with-stderr-to local-erased.ml.stderr + (run %{bin:ocamlformat} --margin-check --erase-jane-syntax %{dep:tests/local.ml}))))) + +(rule + (alias runtest) + (package ocamlformat) + (action (diff tests/local-erased.ml.ref local-erased.ml.stdout))) + +(rule + (alias runtest) + (package ocamlformat) + (action (diff tests/local-erased.ml.err local-erased.ml.stderr))) + +(rule + (deps tests/.ocamlformat ) + (package ocamlformat) + (action + (with-stdout-to local-rewritten.ml.stdout + (with-stderr-to local-rewritten.ml.stderr + (run %{bin:ocamlformat} --margin-check --rewrite-old-style-jane-street-local-annotations %{dep:tests/local.ml}))))) + +(rule + (alias runtest) + (package ocamlformat) + (action (diff tests/local-rewritten.ml.ref local-rewritten.ml.stdout))) + +(rule + (alias runtest) + (package ocamlformat) + (action (diff tests/local-rewritten.ml.err local-rewritten.ml.stderr))) + +(rule + (deps tests/.ocamlformat ) + (package ocamlformat) + (action + (with-stdout-to local.ml.stdout + (with-stderr-to local.ml.stderr + (run %{bin:ocamlformat} --margin-check %{dep:tests/local.ml}))))) + +(rule + (alias runtest) + (package ocamlformat) + (action (diff tests/local.ml.ref local.ml.stdout))) + +(rule + (alias runtest) + (package ocamlformat) + (action (diff tests/local.ml.err local.ml.stderr))) + (rule (deps tests/.ocamlformat ) (package ocamlformat) @@ -3915,6 +4041,24 @@ (package ocamlformat) (action (diff tests/module_type.mli.err module_type.mli.stderr))) +(rule + (deps tests/.ocamlformat ) + (package ocamlformat) + (action + (with-stdout-to module_type_strengthening.ml.stdout + (with-stderr-to module_type_strengthening.ml.stderr + (run %{bin:ocamlformat} --margin-check %{dep:tests/module_type_strengthening.ml}))))) + +(rule + (alias runtest) + (package ocamlformat) + (action (diff tests/module_type_strengthening.ml module_type_strengthening.ml.stdout))) + +(rule + (alias runtest) + (package ocamlformat) + (action (diff tests/module_type_strengthening.ml.err module_type_strengthening.ml.stderr))) + (rule (deps tests/.ocamlformat ) (package ocamlformat) @@ -4276,6 +4420,24 @@ (package ocamlformat) (action (diff tests/parens_tuple_patterns.ml.err parens_tuple_patterns.ml.stderr))) +(rule + (deps tests/.ocamlformat ) + (package ocamlformat) + (action + (with-stdout-to polyparams.ml.stdout + (with-stderr-to polyparams.ml.stderr + (run %{bin:ocamlformat} --margin-check %{dep:tests/polyparams.ml}))))) + +(rule + (alias runtest) + (package ocamlformat) + (action (diff tests/polyparams.ml polyparams.ml.stdout))) + +(rule + (alias runtest) + (package ocamlformat) + (action (diff tests/polyparams.ml.err polyparams.ml.stderr))) + (rule (deps tests/.ocamlformat ) (package ocamlformat) @@ -5290,6 +5452,42 @@ (package ocamlformat) (action (diff tests/unary_hash.ml.err unary_hash.ml.stderr))) +(rule + (deps tests/.ocamlformat ) + (package ocamlformat) + (action + (with-stdout-to unboxed_types.ml.stdout + (with-stderr-to unboxed_types.ml.stderr + (run %{bin:ocamlformat} --margin-check %{dep:tests/unboxed_types.ml}))))) + +(rule + (alias runtest) + (package ocamlformat) + (action (diff tests/unboxed_types.ml unboxed_types.ml.stdout))) + +(rule + (alias runtest) + (package ocamlformat) + (action (diff tests/unboxed_types.ml.err unboxed_types.ml.stderr))) + +(rule + (deps tests/.ocamlformat ) + (package ocamlformat) + (action + (with-stdout-to unboxed_types2.ml.stdout + (with-stderr-to unboxed_types2.ml.stderr + (run %{bin:ocamlformat} --margin-check %{dep:tests/unboxed_types2.ml}))))) + +(rule + (alias runtest) + (package ocamlformat) + (action (diff tests/unboxed_types2.ml.ref unboxed_types2.ml.stdout))) + +(rule + (alias runtest) + (package ocamlformat) + (action (diff tests/unboxed_types2.ml.err unboxed_types2.ml.stderr))) + (rule (deps tests/.ocamlformat ) (package ocamlformat) diff --git a/test/passing/layouts.ml b/test/passing/layouts.ml new file mode 100644 index 0000000000..4fce2a9115 --- /dev/null +++ b/test/passing/layouts.ml @@ -0,0 +1,17 @@ +(* Layout annotations work on type params, including when there are multiple + type params. *) +type ('a : void) t1 = 'a + +type ('b, 'a : immediate) t2 = 'a + +type ('b, 'a : immediate64, 'c) t3 = 'a + +type ('b, 'a : immediate, 'c : any) t4 = 'a + +(* We don't reformat attributes on type parameters to layout annotations + unless there is just one attribute and it's a layout. *) +type 'a[@immediate] [@foo] t5 + +type 'a[@foo] [@immediate] t6 + +type 'a[@baz] t7 diff --git a/test/passing/tests/array.ml b/test/passing/tests/array.ml index 8edf719105..83184284fa 100644 --- a/test/passing/tests/array.ml +++ b/test/passing/tests/array.ml @@ -37,3 +37,45 @@ let f = function ; 1222222 ; 1222222 |] -> () +;; + +(* Immutable arrays *) +[: 1222222 + ; 1222222 + ; 1222222 + ; 1222222 + ; 1222222 + ; 1222222 + ; 1222222 + ; 1222222 + ; 1222222 + ; 1222222 + ; 1222222 + ; 1222222 + ; 1222222 + ; 1222222 + ; 1222222 + ; 1222222 + ; 1222222 + ; 1222222 :] + +let f = function + | [: 1222222 + ; 1222222 + ; 1222222 + ; 1222222 + ; 1222222 + ; 1222222 + ; 1222222 + ; 1222222 + ; 1222222 + ; 1222222 + ; 1222222 + ; 1222222 + ; 1222222 + ; 1222222 + ; 1222222 + ; 1222222 + ; 1222222 + ; 1222222 :] -> + () diff --git a/test/passing/tests/attr_parens.ml b/test/passing/tests/attr_parens.ml new file mode 100644 index 0000000000..60a03f3611 --- /dev/null +++ b/test/passing/tests/attr_parens.ml @@ -0,0 +1,38 @@ +let f = function + | A a -> f a + | B a -> f a [@nontail] + | C a -> f a [@nontail] ; f a [@nontail] + | D a -> + let x = f a [@nontail] in + f x [@nontail] + | E a -> ( + if f a [@nontail] then f a [@nontail] ; + if f a [@nontail] then f a [@nontail] else f a [@nontail] ; + try f a [@nontail] with e -> f a [@nontail] ) + | F a -> new c a [@nontail] + | G a -> + 1 + (f a [@nontail]) ; + 1 ^ (f a [@nontail]) ; + g (f a [@nontail]) ; + 1 :: (f a [@nontail]) + | H a -> + let exception E of string in + f a [@nontail] + | I a -> + let open M in + f a [@nontail] + | J a -> + let module M = N in + f a [@nontail] + | K a -> + let f x = g b [@nontail] in + let f x (type a) = g b [@nontail] in + 42 + | L a -> (f a [@nontail] : t) + | M a -> (f a [@nontail] :> t) + +let f x = g b [@nontail] + +let f x (type a) = g b [@nontail] ;; + +f a [@nontail] diff --git a/test/passing/tests/attributes.ml b/test/passing/tests/attributes.ml index 2d7afb53f6..78e5c79f8c 100644 --- a/test/passing/tests/attributes.ml +++ b/test/passing/tests/attributes.ml @@ -83,7 +83,12 @@ if [@test] true then () else () ;; if [@test] true then () else if [@test] true then () else () -let _ = ((A [@test]), (() [@test]), ([] [@test]), [||] [@test]) +let _ = ((A [@test]), (() [@test]), ([] [@test]), [||] [@test], [::] [@test]) + +let _ = + ( [() for _ = 0 to 1] [@test] + , [|() for _ = 0 to 1|] [@test] + , [:() for _ = 0 to 1:] [@test] ) type blocklist = { f1: int [@version 1, 1, 0] (** short comment *) @@ -144,7 +149,7 @@ let () = () and[@warning "-32"] f = () -external x : a -> b -> (a -> b[@test]) = "" +external x : a -> b -> ((a -> b)[@test]) = "" let f = fun [@test] x y -> () @@ -247,6 +252,14 @@ let _ = f 1 ([e; f] [@a]) let _ = f 1 ([|e; f|] [@a]) +let _ = f 1 ([:e; f:] [@a]) + +let _ = f 1 ([() for _ = 0 to 1] [@a]) + +let _ = f 1 ([|() for _ = 0 to 1|] [@a]) + +let _ = f 1 ([:() for _ = 0 to 1:] [@a]) + let _ = object method g = (a <- b) [@a] @@ -326,6 +339,8 @@ let ([(A | B) [@attr]; b; c] [@attr]) = () let ([|a; (A | B) [@attr]; c|] [@attr]) = () +let ([:a; (A | B) [@attr]; c:] [@attr]) = () + let {b= (A | B) [@attr]} = () let (`Foo ((`A | `B) [@attr])) = () diff --git a/test/passing/tests/attributes.ml.err b/test/passing/tests/attributes.ml.err index 7d1cbc5de8..fd6edd8eef 100644 --- a/test/passing/tests/attributes.ml.err +++ b/test/passing/tests/attributes.ml.err @@ -1 +1 @@ -Warning: tests/attributes.ml:340 exceeds the margin +Warning: tests/attributes.ml:355 exceeds the margin diff --git a/test/passing/tests/break_collection_expressions-wrap.ml.err b/test/passing/tests/break_collection_expressions-wrap.ml.err index 30c5cf20c4..45b258f222 100644 --- a/test/passing/tests/break_collection_expressions-wrap.ml.err +++ b/test/passing/tests/break_collection_expressions-wrap.ml.err @@ -1,2 +1,2 @@ Warning: tests/break_collection_expressions.ml:3 exceeds the margin -Warning: tests/break_collection_expressions.ml:50 exceeds the margin +Warning: tests/break_collection_expressions.ml:72 exceeds the margin diff --git a/test/passing/tests/break_collection_expressions-wrap.ml.ref b/test/passing/tests/break_collection_expressions-wrap.ml.ref index 23018aa057..5f08993c7c 100644 --- a/test/passing/tests/break_collection_expressions-wrap.ml.ref +++ b/test/passing/tests/break_collection_expressions-wrap.ml.ref @@ -18,6 +18,13 @@ let [| fooooooooooooooooooooooooooooooo; fooooooooooooooooooooooooooooooo (* after all elements *) (* after all elements as well *) |] +let [: fooooooooooooooooooooooooooooooo; fooooooooooooooooooooooooooooooo + ; fooooooooooooooooooooooooooooooo (* before end of the array *) :] = + [: fooooooooooooooooooooooooooooooo; fooooooooooooooooooooooooooooooo + ; fooooooooooooooooooooooooooooooo + (* after all elements *) + (* after all elements as well *) :] + let { fooooooooooooooooooooooooooooooo ; fooooooooooooooooooooooooooooooo ; fooooooooooooooooooooooooooooooo @@ -32,6 +39,11 @@ let length = ; (* foo *) 27 (* foo *); 27; 27 |] [@foo] +let length = + [: 0; 269999999999999999999999999999999999999999999999999; 26 + ; (* foo *) 27 (* foo *); 27; 27 :] + [@foo] + let length = [ 0; 14; (* foo *) 14; 17 (* foo *); 17; 2777777777777777777777777777777777 ; 27 ] @@ -47,6 +59,16 @@ let length = ; (* foo *) 27 (* foo *); 27; 27; 27; 27; 27; 27; 27; 27; 27; 28 |] [@foo] +let length = + [: 0; 1; 2; 3; 4; 5; 6; 7; 8; 8; 9; 9; 10; 10; 11; 11; 12; 12; 12; 12; 13 + ; 25; 25; 25; 25; 25; 25; 25; 25; 25; 26; 26; 26; 26; 26; 26; 26; 26; 26 + ; 26; 26; 26; 26; 26; 26 + ; 269999999999999999999999999999999999999999999999999; 26; 26; 26; 26; 26 + ; 26; 26; 26; 26; 26; 26; 26; 26; 26; 26; 26; 27; 27; 27; 27; 27; 27; 27 + ; 27; 27; 27; 27; 27; 27; 27; 27; 27; 27; 27; 27; 27; 27 + ; (* foo *) 27 (* foo *); 27; 27; 27; 27; 27; 27; 27; 27; 27; 28 :] + [@foo] + let length = [ 0; 1; 2; 3; 4; 5; 6; 7; 8; 8; 9; 9; 10; 10; 11; 11; 12; 12; 12; 12; 13; 13 ; 13; 13; 14; 14; 14; (* foo *) 14; 15; 15; 15; 15; 16; 16; 16; 16; 16; 16 diff --git a/test/passing/tests/break_collection_expressions.ml b/test/passing/tests/break_collection_expressions.ml index 1accf32be2..456ab3ee5c 100644 --- a/test/passing/tests/break_collection_expressions.ml +++ b/test/passing/tests/break_collection_expressions.ml @@ -26,6 +26,18 @@ let (* after all elements *) (* after all elements as well *) |] +let + [: fooooooooooooooooooooooooooooooo; + fooooooooooooooooooooooooooooooo; + fooooooooooooooooooooooooooooooo; + (* before end of the array *) :] = + [: fooooooooooooooooooooooooooooooo; + fooooooooooooooooooooooooooooooo; + fooooooooooooooooooooooooooooooo; + (* after all elements *) + (* after all elements as well *) :] + + let { fooooooooooooooooooooooooooooooo; fooooooooooooooooooooooooooooooo; fooooooooooooooooooooooooooooooo; @@ -41,6 +53,11 @@ let length = ; (* foo *) 27 (* foo *); 27; 27 |] [@foo] +let length = + [: 0; 269999999999999999999999999999999999999999999999999; 26 + ; (* foo *) 27 (* foo *); 27; 27 :] + [@foo] + let length = [ 0; 14; (* foo *) 14; 17 (* foo *); 17; 2777777777777777777777777777777777 ; 27 ] @@ -56,6 +73,16 @@ let length = ; (* foo *) 27 (* foo *); 27; 27; 27; 27; 27; 27; 27; 27; 27; 28 |] [@foo] +let length = + [: 0; 1; 2; 3; 4; 5; 6; 7; 8; 8; 9; 9; 10; 10; 11; 11; 12; 12; 12; 12; 13 + ; 25; 25; 25; 25; 25; 25; 25; 25; 25; 26; 26; 26; 26; 26; 26; 26; 26; 26 + ; 26; 26; 26; 26; 26; 26 + ; 269999999999999999999999999999999999999999999999999; 26; 26; 26; 26; 26 + ; 26; 26; 26; 26; 26; 26; 26; 26; 26; 26; 26; 27; 27; 27; 27; 27; 27; 27 + ; 27; 27; 27; 27; 27; 27; 27; 27; 27; 27; 27; 27; 27; 27 + ; (* foo *) 27 (* foo *); 27; 27; 27; 27; 27; 27; 27; 27; 27; 28 :] + [@foo] + let length = [ 0; 1; 2; 3; 4; 5; 6; 7; 8; 8; 9; 9; 10; 10; 11; 11; 12; 12; 12; 12; 13; 13 ; 13; 13; 14; 14; 14; (* foo *) 14; 15; 15; 15; 15; 16; 16; 16; 16; 16; 16 diff --git a/test/passing/tests/break_collection_expressions.ml.ref b/test/passing/tests/break_collection_expressions.ml.ref index fefa7a244e..cb4bc51a4c 100644 --- a/test/passing/tests/break_collection_expressions.ml.ref +++ b/test/passing/tests/break_collection_expressions.ml.ref @@ -22,6 +22,15 @@ let [| fooooooooooooooooooooooooooooooo (* after all elements *) (* after all elements as well *) |] +let [: fooooooooooooooooooooooooooooooo + ; fooooooooooooooooooooooooooooooo + ; fooooooooooooooooooooooooooooooo (* before end of the array *) :] = + [: fooooooooooooooooooooooooooooooo + ; fooooooooooooooooooooooooooooooo + ; fooooooooooooooooooooooooooooooo + (* after all elements *) + (* after all elements as well *) :] + let { fooooooooooooooooooooooooooooooo ; fooooooooooooooooooooooooooooooo ; fooooooooooooooooooooooooooooooo @@ -40,6 +49,15 @@ let length = ; 27 |] [@foo] +let length = + [: 0 + ; 269999999999999999999999999999999999999999999999999 + ; 26 + ; (* foo *) 27 (* foo *) + ; 27 + ; 27 :] + [@foo] + let length = [ 0 ; 14 @@ -148,6 +166,103 @@ let length = ; 28 |] [@foo] +let length = + [: 0 + ; 1 + ; 2 + ; 3 + ; 4 + ; 5 + ; 6 + ; 7 + ; 8 + ; 8 + ; 9 + ; 9 + ; 10 + ; 10 + ; 11 + ; 11 + ; 12 + ; 12 + ; 12 + ; 12 + ; 13 + ; 25 + ; 25 + ; 25 + ; 25 + ; 25 + ; 25 + ; 25 + ; 25 + ; 25 + ; 26 + ; 26 + ; 26 + ; 26 + ; 26 + ; 26 + ; 26 + ; 26 + ; 26 + ; 26 + ; 26 + ; 26 + ; 26 + ; 26 + ; 26 + ; 269999999999999999999999999999999999999999999999999 + ; 26 + ; 26 + ; 26 + ; 26 + ; 26 + ; 26 + ; 26 + ; 26 + ; 26 + ; 26 + ; 26 + ; 26 + ; 26 + ; 26 + ; 26 + ; 26 + ; 27 + ; 27 + ; 27 + ; 27 + ; 27 + ; 27 + ; 27 + ; 27 + ; 27 + ; 27 + ; 27 + ; 27 + ; 27 + ; 27 + ; 27 + ; 27 + ; 27 + ; 27 + ; 27 + ; 27 + ; 27 + ; (* foo *) 27 (* foo *) + ; 27 + ; 27 + ; 27 + ; 27 + ; 27 + ; 27 + ; 27 + ; 27 + ; 27 + ; 28 :] + [@foo] + let length = [ 0 ; 1 diff --git a/test/passing/tests/break_separators-after.ml.ref b/test/passing/tests/break_separators-after.ml.ref index 5353f8eef8..4fe3e2ed66 100644 --- a/test/passing/tests/break_separators-after.ml.ref +++ b/test/passing/tests/break_separators-after.ml.ref @@ -89,6 +89,14 @@ let _ = 0 | _ -> 1 +let _ = + match something with + | [: very_very_long_field_name_running_out_of_space; + another_very_very_long_field_name_running_out_of_space; + _ :] -> + 0 + | _ -> 1 + [@@@ocamlformat "type-decl=compact"] type t = {aaaaaaaaa: aaaa; bbbbbbbbb: bbbb} @@ -144,6 +152,16 @@ let length = 27 |] [@foo] +(* this is an immutable array *) +let length = + [: 0; + 269999999999999999999999999999999999999999999999999; + 26; + (* foo *) 27 (* foo *); + 27; + 27 :] + [@foo] + (* this is a list *) let length = [ 0; @@ -155,6 +173,36 @@ let length = 2777777777777777777777777777777777; 27 ] [@foo] + +(* Comprehensions are invariant under separator placement and wrapping vs. + breaking, but respect delimiter docking behavior *) + +(* this is a list comprehension *) +let pythagorean = + [ (a, b, c) + for a = 1 to 10 + for b = a to 10 + for c = b to 10 + when (a * a) + (b * b) = c * c ] + [@foo] + +(* this is an array comprehension *) +let pythagorean = + [| (a, b, c) + for a = 1 to 10 + for b = a to 10 + for c = b to 10 + when (a * a) + (b * b) = c * c |] + [@foo] + +(* this is an immutable array comprehension *) +let pythagorean = + [: (a, b, c) + for a = 1 to 10 + for b = a to 10 + for c = b to 10 + when (a * a) + (b * b) = c * c :] + [@foo] ;; Fooooooo.foo ~foooooooooooooo ~fooooooooo:"" @@ -341,6 +389,20 @@ let f () = in foooooooooooo +let f () = + let [: aaaaaaaa; + bbbbbbbbbb; + ccccccccc; + dddddddddd; + eeeeeeeee; + ffffffffff; + gggggggggg; + hhhhhhhhhh :] + = + some_value + in + foooooooooooo + let g () = match some_value with | { aaaaaaaa; @@ -370,6 +432,15 @@ let g () = gggggggggg; hhhhhhhhhh |] -> fooooooooo + | [: aaaaaaaa; + bbbbbbbbbb; + ccccccccc; + dddddddddd; + eeeeeeeee; + ffffffffff; + gggggggggg; + hhhhhhhhhh :] -> + fooooooooo let () = match x with _, (* line 1 line 2 *) Some _ -> x diff --git a/test/passing/tests/break_separators-after_docked.ml.err b/test/passing/tests/break_separators-after_docked.ml.err index 2ccd970c7d..49362248e6 100644 --- a/test/passing/tests/break_separators-after_docked.ml.err +++ b/test/passing/tests/break_separators-after_docked.ml.err @@ -1 +1 @@ -Warning: tests/break_separators.ml:335 exceeds the margin +Warning: tests/break_separators.ml:393 exceeds the margin diff --git a/test/passing/tests/break_separators-after_docked.ml.ref b/test/passing/tests/break_separators-after_docked.ml.ref index ae435ee3a2..b1ab6b3c08 100644 --- a/test/passing/tests/break_separators-after_docked.ml.ref +++ b/test/passing/tests/break_separators-after_docked.ml.ref @@ -100,6 +100,16 @@ let _ = 0 | _ -> 1 +let _ = + match something with + | [: + very_very_long_field_name_running_out_of_space; + another_very_very_long_field_name_running_out_of_space; + _; + :] -> + 0 + | _ -> 1 + [@@@ocamlformat "type-decl=compact"] type t = {aaaaaaaaa: aaaa; bbbbbbbbb: bbbb} @@ -165,6 +175,18 @@ let length = |] [@foo] +(* this is an immutable array *) +let length = + [: + 0; + 269999999999999999999999999999999999999999999999999; + 26; + (* foo *) 27 (* foo *); + 27; + 27; + :] + [@foo] + (* this is a list *) let length = [ @@ -178,6 +200,42 @@ let length = 27; ] [@foo] + +(* Comprehensions are invariant under separator placement and wrapping vs. + breaking, but respect delimiter docking behavior *) + +(* this is a list comprehension *) +let pythagorean = + [ + (a, b, c) + for a = 1 to 10 + for b = a to 10 + for c = b to 10 + when (a * a) + (b * b) = c * c + ] + [@foo] + +(* this is an array comprehension *) +let pythagorean = + [| + (a, b, c) + for a = 1 to 10 + for b = a to 10 + for c = b to 10 + when (a * a) + (b * b) = c * c + |] + [@foo] + +(* this is an immutable array comprehension *) +let pythagorean = + [: + (a, b, c) + for a = 1 to 10 + for b = a to 10 + for c = b to 10 + when (a * a) + (b * b) = c * c + :] + [@foo] ;; Fooooooo.foo ~foooooooooooooo ~fooooooooo:"" @@ -384,6 +442,22 @@ let f () = in foooooooooooo +let f () = + let [: + aaaaaaaa; + bbbbbbbbbb; + ccccccccc; + dddddddddd; + eeeeeeeee; + ffffffffff; + gggggggggg; + hhhhhhhhhh; + :] + = + some_value + in + foooooooooooo + let g () = match some_value with | { @@ -419,6 +493,17 @@ let g () = hhhhhhhhhh; |] -> fooooooooo + | [: + aaaaaaaa; + bbbbbbbbbb; + ccccccccc; + dddddddddd; + eeeeeeeee; + ffffffffff; + gggggggggg; + hhhhhhhhhh; + :] -> + fooooooooo let () = match x with _, (* line 1 line 2 *) Some _ -> x diff --git a/test/passing/tests/break_separators-before_docked.ml.ref b/test/passing/tests/break_separators-before_docked.ml.ref index 7d0f75e25e..65d38e1acf 100644 --- a/test/passing/tests/break_separators-before_docked.ml.ref +++ b/test/passing/tests/break_separators-before_docked.ml.ref @@ -100,6 +100,16 @@ let _ = 0 | _ -> 1 +let _ = + match something with + | [: + very_very_long_field_name_running_out_of_space + ; another_very_very_long_field_name_running_out_of_space + ; _ + :] -> + 0 + | _ -> 1 + [@@@ocamlformat "type-decl=compact"] type t = {aaaaaaaaa: aaaa; bbbbbbbbb: bbbb} @@ -165,6 +175,18 @@ let length = |] [@foo] +(* this is an immutable array *) +let length = + [: + 0 + ; 269999999999999999999999999999999999999999999999999 + ; 26 + ; (* foo *) 27 (* foo *) + ; 27 + ; 27 + :] + [@foo] + (* this is a list *) let length = [ @@ -178,6 +200,42 @@ let length = ; 27 ] [@foo] + +(* Comprehensions are invariant under separator placement and wrapping vs. + breaking, but respect delimiter docking behavior *) + +(* this is a list comprehension *) +let pythagorean = + [ + (a, b, c) + for a = 1 to 10 + for b = a to 10 + for c = b to 10 + when (a * a) + (b * b) = c * c + ] + [@foo] + +(* this is an array comprehension *) +let pythagorean = + [| + (a, b, c) + for a = 1 to 10 + for b = a to 10 + for c = b to 10 + when (a * a) + (b * b) = c * c + |] + [@foo] + +(* this is an immutable array comprehension *) +let pythagorean = + [: + (a, b, c) + for a = 1 to 10 + for b = a to 10 + for c = b to 10 + when (a * a) + (b * b) = c * c + :] + [@foo] ;; Fooooooo.foo ~foooooooooooooo ~fooooooooo:"" @@ -384,6 +442,22 @@ let f () = in foooooooooooo +let f () = + let [: + aaaaaaaa + ; bbbbbbbbbb + ; ccccccccc + ; dddddddddd + ; eeeeeeeee + ; ffffffffff + ; gggggggggg + ; hhhhhhhhhh + :] + = + some_value + in + foooooooooooo + let g () = match some_value with | { @@ -419,6 +493,17 @@ let g () = ; hhhhhhhhhh |] -> fooooooooo + | [: + aaaaaaaa + ; bbbbbbbbbb + ; ccccccccc + ; dddddddddd + ; eeeeeeeee + ; ffffffffff + ; gggggggggg + ; hhhhhhhhhh + :] -> + fooooooooo let () = match x with _, (* line 1 line 2 *) Some _ -> x diff --git a/test/passing/tests/break_separators.ml b/test/passing/tests/break_separators.ml index 29a972901e..60639b2fdb 100644 --- a/test/passing/tests/break_separators.ml +++ b/test/passing/tests/break_separators.ml @@ -89,6 +89,14 @@ let _ = 0 | _ -> 1 +let _ = + match something with + | [: very_very_long_field_name_running_out_of_space + ; another_very_very_long_field_name_running_out_of_space + ; _ :] -> + 0 + | _ -> 1 + [@@@ocamlformat "type-decl=compact"] type t = {aaaaaaaaa: aaaa; bbbbbbbbb: bbbb} @@ -144,6 +152,16 @@ let length = ; 27 |] [@foo] +(* this is an immutable array *) +let length = + [: 0 + ; 269999999999999999999999999999999999999999999999999 + ; 26 + ; (* foo *) 27 (* foo *) + ; 27 + ; 27 :] + [@foo] + (* this is a list *) let length = [ 0 @@ -155,6 +173,36 @@ let length = ; 2777777777777777777777777777777777 ; 27 ] [@foo] + +(* Comprehensions are invariant under separator placement and wrapping vs. + breaking, but respect delimiter docking behavior *) + +(* this is a list comprehension *) +let pythagorean = + [ (a, b, c) + for a = 1 to 10 + for b = a to 10 + for c = b to 10 + when (a * a) + (b * b) = c * c ] + [@foo] + +(* this is an array comprehension *) +let pythagorean = + [| (a, b, c) + for a = 1 to 10 + for b = a to 10 + for c = b to 10 + when (a * a) + (b * b) = c * c |] + [@foo] + +(* this is an immutable array comprehension *) +let pythagorean = + [: (a, b, c) + for a = 1 to 10 + for b = a to 10 + for c = b to 10 + when (a * a) + (b * b) = c * c :] + [@foo] ;; Fooooooo.foo ~foooooooooooooo ~fooooooooo:"" @@ -341,6 +389,20 @@ let f () = in foooooooooooo +let f () = + let [: aaaaaaaa + ; bbbbbbbbbb + ; ccccccccc + ; dddddddddd + ; eeeeeeeee + ; ffffffffff + ; gggggggggg + ; hhhhhhhhhh :] + = + some_value + in + foooooooooooo + let g () = match some_value with | { aaaaaaaa @@ -370,6 +432,15 @@ let g () = ; gggggggggg ; hhhhhhhhhh |] -> fooooooooo + | [: aaaaaaaa + ; bbbbbbbbbb + ; ccccccccc + ; dddddddddd + ; eeeeeeeee + ; ffffffffff + ; gggggggggg + ; hhhhhhhhhh :] -> + fooooooooo let () = match x with _, (* line 1 line 2 *) Some _ -> x diff --git a/test/passing/tests/comment_in_empty.ml b/test/passing/tests/comment_in_empty.ml index 6d5cda8ed2..b02e3b25e8 100644 --- a/test/passing/tests/comment_in_empty.ml +++ b/test/passing/tests/comment_in_empty.ml @@ -18,6 +18,8 @@ let _ = (* this list is empty2 *) [] let _ = [| (* this array is empty *) |] +let _ = [: (* this immutable array is empty *) :] + let _ = f ( (* comment in unit *) ) let _ = f "asd" (* te""st *) 3 @@ -25,6 +27,7 @@ let _ = f "asd" (* te""st *) 3 let x = function | [ (* empty list pat *) ] |[| (* empty array pat *) |] + |[: (* empty immutable array pat *) :] |( (* unit pat *) ) |"" (* comment *) -> () diff --git a/test/passing/tests/comprehensions.ml b/test/passing/tests/comprehensions.ml new file mode 100644 index 0000000000..fde31ddba8 --- /dev/null +++ b/test/passing/tests/comprehensions.ml @@ -0,0 +1,224 @@ +(********************************************************************* + * Lists *) + +(* Pythagorean triples with components from 1 to 10, no duplicate triples *) +let pyth = + [ (a, b, c) + for a = 1 to 10 + for b = a to 10 + for c = b to 10 + when (a * a) + (b * b) = c * c ] + +(* Let's describe some objects *) +let descriptions = + [ Printf.sprintf "a %s %s" adjective noun + for noun in ["light"; "pepper"] + and adjective in ["red"; "yellow"; "green"] ] + +(* Compute a list of reciprocals in increasing order *) +let reciprocals = [1. /. Float.of_int x for x = 5 downto 0 when x <> 0] + +(* Flatten a nested array *) +let flattened = + let sentences = + [ ["hello"; "world"] + ; ["how"; "are"; "you"; "doing"] + ; ["please"; "enjoy"; "these"; "comprehensions"] ] + in + [word for sentence in sentences for word in sentence] + +(* We could use comprehensions to reimplement map... *) +let map' ~f l = [f x for x in l] + +(* ...and filter *) +let filter' ~f l = [x for x in l when f x] + +(* We can put comprehensions inside other comprehensions... *) +let nested_comprehensions = [[i for i = 1 to n] for n = 1 to 10] + +(* ...in multiple_places *) +let nested_comprehensions_rhs = + [k for n in [i * i for i = 1 to 10] for k = 1 to n] + +(* Sometimes comprehensions need to line-wrap *) +let wrapping_inside_individual_comprehension_pieces = + [ this is a very long function_application so that we can see how the body + of_a comprehension line wraps + for + thoroughness = + we also want to_know how line wrapping looks inside the right hand + side of_a clause downto its length + and + similarly = we want + to know how line wrapping looks for_things_in every single part_of a + clause + for + example in + the sequence iteration case we also want_to test the line wrapping + behavior + when + we have a conditional we also want_to test the line wrapping behavior + for_it + for + some_patterns_can_get_so_long_that_after_them_they_force_wrapping = 0 + to 1 + and + those_patterns_themselves_are_long_enough_that_they_can't_break in + -ternally + and + can_even_grow_long_enough_to_force_wrapping_to_occur_right_before_the + in keyword + for + ( Other (patterns, they, can, get, really, terribly) + , Excessively (long, so, that, they, need, line, breaking) ) in + their own right ] + +(********************************************************************* + * Arrays *) + +(* Pythagorean triples with components from 1 to 10, no duplicate triples *) +let pyth = + [| (a, b, c) + for a = 1 to 10 + for b = a to 10 + for c = b to 10 + when (a * a) + (b * b) = c * c |] + +(* Let's describe some objects *) +let descriptions = + [| Printf.sprintf "a %s %s" adjective noun + for noun in [|"light"; "pepper"|] + and adjective in [|"red"; "yellow"; "green"|] |] + +(* Compute a list of reciprocals in increasing order *) +let reciprocals = [|1. /. Float.of_int x for x = 5 downto 0 when x <> 0|] + +(* Flatten a nested array *) +let flattened = + let sentences = + [| [|"hello"; "world"|] + ; [|"how"; "are"; "you"; "doing"|] + ; [|"please"; "enjoy"; "these"; "comprehensions"|] |] + in + [|word for sentence in sentences for word in sentence|] + +(* We could use comprehensions to reimplement map... *) +let map' ~f l = [|f x for x in l|] + +(* ...and filter *) +let filter' ~f l = [|x for x in l when f x|] + +(* We can put comprehensions inside other comprehensions... *) +let nested_comprehensions = [|[|i for i = 1 to n|] for n = 1 to 10|] + +(* ...in multiple_places *) +let nested_comprehensions_rhs = + [|k for n in [|i * i for i = 1 to 10|] for k = 1 to n|] + +(* Sometimes comprehensions need to line-wrap *) +let wrapping_inside_individual_comprehension_pieces = + [| this is a very long function_application so that we can see how the body + of_a comprehension line wraps + for + thoroughness = + we also want to_know how line wrapping looks inside the right hand + side of_a clause downto its length + and + similarly = we want + to know how line wrapping looks for_things_in every single part_of a + clause + for + example in + the sequence iteration case we also want_to test the line wrapping + behavior + when + we have a conditional we also want_to test the line wrapping behavior + for_it + for + some_patterns_can_get_so_long_that_after_them_they_force_wrapping = 0 + to 1 + and + those_patterns_themselves_are_long_enough_that_they_can't_break in + -ternally + and + can_even_grow_long_enough_to_force_wrapping_to_occur_right_before_the + in keyword + for + ( Other (patterns, they, can, get, really, terribly) + , Excessively (long, so, that, they, need, line, breaking) ) in + their own right |] + +(********************************************************************* + * Immutable arrays *) + +(* Pythagorean triples with components from 1 to 10, no duplicate triples *) +let pyth = + [: (a, b, c) + for a = 1 to 10 + for b = a to 10 + for c = b to 10 + when (a * a) + (b * b) = c * c :] + +(* Let's describe some objects *) +let descriptions = + [: Printf.sprintf "a %s %s" adjective noun + for noun in [:"light"; "pepper":] + and adjective in [:"red"; "yellow"; "green":] :] + +(* Compute a list of reciprocals in increasing order *) +let reciprocals = [:1. /. Float.of_int x for x = 5 downto 0 when x <> 0:] + +(* Flatten a nested array *) +let flattened = + let sentences = + [: [:"hello"; "world":] + ; [:"how"; "are"; "you"; "doing":] + ; [:"please"; "enjoy"; "these"; "comprehensions":] :] + in + [:word for sentence in sentences for word in sentence:] + +(* We could use comprehensions to reimplement map... *) +let map' ~f l = [:f x for x in l:] + +(* ...and filter *) +let filter' ~f l = [:x for x in l when f x:] + +(* We can put comprehensions inside other comprehensions... *) +let nested_comprehensions = [:[:i for i = 1 to n:] for n = 1 to 10:] + +(* ...in multiple_places *) +let nested_comprehensions_rhs = + [:k for n in [:i * i for i = 1 to 10:] for k = 1 to n:] + +(* Sometimes comprehensions need to line-wrap *) +let wrapping_inside_individual_comprehension_pieces = + [: this is a very long function_application so that we can see how the body + of_a comprehension line wraps + for + thoroughness = + we also want to_know how line wrapping looks inside the right hand + side of_a clause downto its length + and + similarly = we want + to know how line wrapping looks for_things_in every single part_of a + clause + for + example in + the sequence iteration case we also want_to test the line wrapping + behavior + when + we have a conditional we also want_to test the line wrapping behavior + for_it + for + some_patterns_can_get_so_long_that_after_them_they_force_wrapping = 0 + to 1 + and + those_patterns_themselves_are_long_enough_that_they_can't_break in + -ternally + and + can_even_grow_long_enough_to_force_wrapping_to_occur_right_before_the + in keyword + for + ( Other (patterns, they, can, get, really, terribly) + , Excessively (long, so, that, they, need, line, breaking) ) in + their own right :] diff --git a/test/passing/tests/exp_grouping-parens.ml.ref b/test/passing/tests/exp_grouping-parens.ml.ref index fa3e31470c..4dadb5a1b1 100644 --- a/test/passing/tests/exp_grouping-parens.ml.ref +++ b/test/passing/tests/exp_grouping-parens.ml.ref @@ -204,6 +204,10 @@ let _ = [| (let a = b in c ) |] +let _ = + [: (let a = b in + c ) :] + let () = if a then b (* asd *) [@@@ocamlformat "if-then-else=compact"] diff --git a/test/passing/tests/exp_grouping.ml b/test/passing/tests/exp_grouping.ml index 325a250ea0..c98f132a41 100644 --- a/test/passing/tests/exp_grouping.ml +++ b/test/passing/tests/exp_grouping.ml @@ -121,6 +121,7 @@ let _ = let a = b in c :: d let _ = a :: ( let a = b in c ) let _ = [ ( let a = b in c ) ] let _ = [| ( let a = b in c ) |] +let _ = [: ( let a = b in c ) :] let () = if a then begin b diff --git a/test/passing/tests/exp_grouping.ml.ref b/test/passing/tests/exp_grouping.ml.ref index 14493e5665..459311ab15 100644 --- a/test/passing/tests/exp_grouping.ml.ref +++ b/test/passing/tests/exp_grouping.ml.ref @@ -214,6 +214,10 @@ let _ = [| (let a = b in c ) |] +let _ = + [: (let a = b in + c ) :] + let () = if a then begin b (* asd *) diff --git a/test/passing/tests/immutable_arrays.ml b/test/passing/tests/immutable_arrays.ml new file mode 100644 index 0000000000..f7c06d75d8 --- /dev/null +++ b/test/passing/tests/immutable_arrays.ml @@ -0,0 +1,11 @@ +let expression = [:1; 2; 3:] + +let pattern = function [::] -> 0 | [:x:] -> x | [:_; _:] -> 2 | _ -> -1 + +let nested = [:[:1; 2:]; [:3; 4:]; [:5; 6; 7:]:] + +let nested_pattern [:[:1; 2:]; [:3; 4:]; [:5; 6; 7:]:] = () + +let indexed_literal = [:"a"; "b"; "c":].:(0) + +let indexed_variable iarray = iarray.:(1) diff --git a/test/passing/tests/include_functor.ml b/test/passing/tests/include_functor.ml new file mode 100644 index 0000000000..6fa94df23b --- /dev/null +++ b/test/passing/tests/include_functor.ml @@ -0,0 +1,38 @@ +(* examples taken from: + ocaml/testsuite/tests/typing-modules/include_functor.ml *) + +(* In structure *) +module type S = sig + type t + + val x : t +end + +module F1 (X : S) = struct + let y = X.x +end + +module M1 = struct + type t = int + + let x = 5 + + include functor F1 +end + +(* In signature *) +module type T = sig + type s + + val f : s -> bool +end + +module type F5 = functor (X : S) -> T with type s = X.t + +module type M5_sig = sig + type t + + val x : t + + include functor F5 +end diff --git a/test/passing/tests/index_op.ml b/test/passing/tests/index_op.ml index d4c8636cc1..20fe6ac91e 100644 --- a/test/passing/tests/index_op.ml +++ b/test/passing/tests/index_op.ml @@ -175,6 +175,12 @@ let _ = Array.get [||] (-8) let _ = Array.unsafe_get [||] (-8) +let _ = [::].:(-8) + +let _ = Iarray.get [::] (-8) + +let _ = Iarray.unsafe_get [::] (-8) + let _ = Bigarray.Genarray.get x [||] (-8) let _ = Bigarray.Genarray.unsafe_get x [||] (-8) diff --git a/test/passing/tests/js_source.ml.ocp b/test/passing/tests/js_source.ml.ocp index 1c092ebdc0..7863961cd5 100644 --- a/test/passing/tests/js_source.ml.ocp +++ b/test/passing/tests/js_source.ml.ocp @@ -2448,7 +2448,7 @@ let intAorB = function type _ wrapPoly = WrapPoly : 'a poly -> ([< `TagA of int | `TagB ] as 'a) wrapPoly -let example6 : type a. a wrapPoly -> a -> int = +let example6 : type a. a wrapPoly -> (a -> int) = fun w -> match w with | WrapPoly ATag -> intA diff --git a/test/passing/tests/js_source.ml.ref b/test/passing/tests/js_source.ml.ref index bc74fe3a0f..dd4cb8b407 100644 --- a/test/passing/tests/js_source.ml.ref +++ b/test/passing/tests/js_source.ml.ref @@ -2448,7 +2448,7 @@ let intAorB = function type _ wrapPoly = WrapPoly : 'a poly -> ([< `TagA of int | `TagB ] as 'a) wrapPoly -let example6 : type a. a wrapPoly -> a -> int = +let example6 : type a. a wrapPoly -> (a -> int) = fun w -> match w with | WrapPoly ATag -> intA diff --git a/test/passing/tests/label_option_default_args.ml b/test/passing/tests/label_option_default_args.ml index bca109bbdf..b8e8a3b234 100644 --- a/test/passing/tests/label_option_default_args.ml +++ b/test/passing/tests/label_option_default_args.ml @@ -87,6 +87,8 @@ let f ?record:({a; b}) = () let f ?array:([| 1; 2; 3 |]) = () +let f ?iarray:([: 1; 2; 3 :]) = () + let f ?or_:(Some () | None) = () let f ?constraint_:(() : unit) = () diff --git a/test/passing/tests/label_option_default_args.ml.ref b/test/passing/tests/label_option_default_args.ml.ref index 09efc00fa6..cf2a6d736f 100644 --- a/test/passing/tests/label_option_default_args.ml.ref +++ b/test/passing/tests/label_option_default_args.ml.ref @@ -9,9 +9,8 @@ let (* 0 *) f ~(* 1 *) x (* 2 *) = (* 3 *) e let f ~(x : t) = e let (* 0 *) f - ~(* 1 *) - (* 2 *) - (x (* 3 *) : (* 4 *) t (* 5 *)) = + ~((* 1 *) + (* 2 *) x (* 3 *) : (* 4 *) t (* 5 *) ) = (* 6 *) (* 7 *) e @@ -48,9 +47,8 @@ let (* 0 *) f let f ?(x : t) = e let (* 0 *) f - ?(* 1 *) - (* 2 *) - (x (* 3 *) : (* 4 *) t (* 5 *)) = + ?((* 1 *) + (* 2 *) x (* 3 *) : (* 4 *) t (* 5 *) ) = (* 6 *) e let f ?l:x = e @@ -142,6 +140,8 @@ let f ?record:({a; b}) = () let f ?array:([|1; 2; 3|]) = () +let f ?iarray:([:1; 2; 3:]) = () + let f ?or_:(Some () | None) = () let f ?constraint_:(() : unit) = () diff --git a/test/passing/tests/line_directives.ml.err b/test/passing/tests/line_directives.ml.err index 47533e6a3f..5ed00e0a62 100644 --- a/test/passing/tests/line_directives.ml.err +++ b/test/passing/tests/line_directives.ml.err @@ -1,5 +1,5 @@ ocamlformat: ignoring "tests/line_directives.ml" (syntax error) -File "tests/line_directives.ml", line 1, characters 1-9: +File "tests/line_directives.ml", line 1, characters 0-9: 1 | #3 "f.ml" - ^^^^^^^^ + ^^^^^^^^^ Error: Invalid lexer directive "#3 \"f.ml\"": line directives are not supported diff --git a/test/passing/tests/local-erased.ml.opts b/test/passing/tests/local-erased.ml.opts new file mode 100644 index 0000000000..a7fb969009 --- /dev/null +++ b/test/passing/tests/local-erased.ml.opts @@ -0,0 +1 @@ +--erase-jane-syntax diff --git a/test/passing/tests/local-erased.ml.ref b/test/passing/tests/local-erased.ml.ref new file mode 100644 index 0000000000..fc552815b5 --- /dev/null +++ b/test/passing/tests/local-erased.ml.ref @@ -0,0 +1,127 @@ +let f a b c = 1 + +let f a ~foo:b ?foo:(c = 1) ~d = () + +let f ~x ~(y : string) ?(z : string) = () + +let xs = [(fun a (type b) ~c -> 1)] + +let xs = [(fun a (type b) ~c -> 1)] + +let f () = + let a = [1] in + let r = 1 in + let f : 'a. 'a -> 'a = fun x -> x in + let g a b c : int = 1 in + let () = g (fun () -> ()) in + "asdfasdfasdfasdfasdfasdfasdf" + +let f () = + let a = [1] in + let r = 1 in + let f : 'a. 'a -> 'a = fun x -> x in + let g a b c : int = 1 in + let () = g (fun () -> ()) in + "asdfasdfasdfasdfasdfasdfasdf" + +type 'a r = {mutable a: 'a; b: 'a; c: 'a} + +type 'a r = Foo of 'a | Bar of 'a * 'a | Baz of int * string * 'a + +type ('a, 'b) cfn = a:'a -> ?b:b -> 'a -> int -> 'b + +let _ = () + +let _ = () + +let () = x + +let () = x + +let {b} = () + +let {b} = () + +let () = r + +let () = r + +let x : string = "hi" + +let (x : string) = "hi" + +let (x : string) = "hi" + +let x = ("hi" : string) + +let x = ("hi" : string) + +let x : 'a. 'a -> 'a = "hi" + +let x : 'a. 'a -> 'a = "hi" + +let f : 'a. 'a -> 'a = "hi" + +let foo () = + if true then () ; + () + +type loc_long_attrs = (string[@ocaml.local]) -> (string[@ocaml.local]) + +type loc_short_attrs = (string[@local]) -> (string[@local]) + +type global_long_attrs = + | Foo of {s: string [@ocaml.global]} + | Bar of (string[@ocaml.global]) + +type global_short_attrs = + | Foo of {s: string [@global]} + | Bar of (string[@global]) + +type global_short_attrs = + | Foo of {s: string [@global]} + | Bar of (string[@global]) + +let local_long_ext = [%ocaml.local] () + +let local_short_ext = [%local] () + +let exclave_long_ext = [%ocaml.exclave] () + +let exclave_short_ext = [%exclave] () + +let[@ocaml.local] upstream_local_attr_long x = x + +let[@ocaml.local never] upstream_local_attr_never_long x = x + +let[@ocaml.local always] upstream_local_attr_always_long x = x + +let[@ocaml.local maybe] upstream_local_attr_maybe_long x = x + +let[@local] upstream_local_attr_short x = x + +let[@local never] upstream_local_attr_never_short x = x + +let[@local always] upstream_local_attr_always_short x = x + +let[@local maybe] upstream_local_attr_maybe_short x = x + +let f x = + (* a *) + let y = 1 in + x + y + +let f x = + (* a *) + let y = 1 in + x + y + +let x = + (* a *) + let y = 1 in + y + +let x = + (* a *) + let y = 1 in + y diff --git a/test/passing/tests/local-rewritten.ml.opts b/test/passing/tests/local-rewritten.ml.opts new file mode 100644 index 0000000000..11c5f41118 --- /dev/null +++ b/test/passing/tests/local-rewritten.ml.opts @@ -0,0 +1 @@ +--rewrite-old-style-jane-street-local-annotations diff --git a/test/passing/tests/local-rewritten.ml.ref b/test/passing/tests/local-rewritten.ml.ref new file mode 100644 index 0000000000..87594a5ad6 --- /dev/null +++ b/test/passing/tests/local-rewritten.ml.ref @@ -0,0 +1,133 @@ +let f a b c = 1 + +let f (local_ a) ~foo:(local_ b) ?foo:(local_ c = 1) ~(local_ d) = () + +let f ~(local_ x) ~(local_ y : string) ?(local_ z : string) = () + +let xs = [(fun (local_ a) (type b) ~(local_ c) -> local_ 1)] + +let xs = [(fun (local_ a) (type b) ~(local_ c) -> exclave_ 1)] + +let f () = local_ + let a = [local_ 1] in + let local_ r = 1 in + let local_ f : 'a. 'a -> 'a = fun x -> local_ x in + let local_ g a b c : int = 1 in + let () = g (local_ fun () -> ()) in + local_ "asdfasdfasdfasdfasdfasdfasdf" + +let f () = exclave_ + let a = [exclave_ 1] in + let local_ r = 1 in + let local_ f : 'a. 'a -> 'a = fun x -> exclave_ x in + let local_ g a b c : int = 1 in + let () = g (exclave_ fun () -> ()) in + exclave_ "asdfasdfasdfasdfasdfasdfasdf" + +type 'a r = {mutable a: 'a; b: 'a; global_ c: 'a} + +type 'a r = + | Foo of global_ 'a + | Bar of 'a * global_ 'a + | Baz of global_ int * string * global_ 'a + +type ('a, 'b) cfn = + a:local_ 'a -> ?b:local_ b -> local_ 'a -> (int -> local_ 'b) + +let _ = local_ () + +let _ = exclave_ () + +let () = local_ x + +let () = exclave_ x + +let {b} = local_ () + +let {b} = exclave_ () + +let () = local_ r + +let () = exclave_ r + +let local_ x : string = "hi" + +let (x : string) = local_ "hi" + +let (x : string) = exclave_ "hi" + +let local_ x = ("hi" : string) + +let x = exclave_ ("hi" : string) + +let x : 'a. 'a -> 'a = local_ "hi" + +let x : 'a. 'a -> 'a = exclave_ "hi" + +let local_ f : 'a. 'a -> 'a = "hi" + +let foo () = + if true then (local_ ()) ; + () + +type loc_long_attrs = local_ string -> local_ string + +type loc_short_attrs = local_ string -> local_ string + +type global_long_attrs = Foo of {global_ s: string} | Bar of global_ string + +type global_short_attrs = + | Foo of {global_ s: string} + | Bar of global_ string + +type global_short_attrs = + | Foo of {global_ s: string} + | Bar of global_ string + +let local_ local_long_ext = () + +let local_ local_short_ext = () + +let exclave_long_ext = exclave_ () + +let exclave_short_ext = exclave_ () + +let[@ocaml.local] upstream_local_attr_long x = x + +let[@ocaml.local never] upstream_local_attr_never_long x = x + +let[@ocaml.local always] upstream_local_attr_always_long x = x + +let[@ocaml.local maybe] upstream_local_attr_maybe_long x = x + +let[@local] upstream_local_attr_short x = x + +let[@local never] upstream_local_attr_never_short x = x + +let[@local always] upstream_local_attr_always_short x = x + +let[@local maybe] upstream_local_attr_maybe_short x = x + +let f x = + (* a *) + local_ + let y = 1 in + x + y + +let f x = + (* a *) + exclave_ + let y = 1 in + x + y + +let x = + (* a *) + local_ + let y = 1 in + y + +let x = + (* a *) + exclave_ + let y = 1 in + y diff --git a/test/passing/tests/local.ml b/test/passing/tests/local.ml new file mode 100644 index 0000000000..c8af438a72 --- /dev/null +++ b/test/passing/tests/local.ml @@ -0,0 +1,123 @@ +let f a b c = 1 + +let f (local_ a) ~foo:(local_ b) ?foo:(local_ c = 1) ~(local_ d) = () + +let f ~(local_ x) ~(local_ y : string) ?(local_ z : string) = () + +let xs = [(fun (local_ a) (type b) ~(local_ c) -> local_ 1)] + +let xs = [(fun (local_ a) (type b) ~(local_ c) -> exclave_ 1)] + +let f () = local_ + let a = [local_ 1] in + let local_ r = 1 in + let local_ f : 'a. 'a -> 'a = fun x -> local_ x in + let local_ g a b c : int = 1 in + let () = g (local_ fun () -> ()) in + local_ "asdfasdfasdfasdfasdfasdfasdf" + +let f () = exclave_ + let a = [exclave_ 1] in + let local_ r = 1 in + let local_ f : 'a. 'a -> 'a = fun x -> exclave_ x in + let local_ g a b c : int = 1 in + let () = g (exclave_ (fun () -> ())) in + exclave_ "asdfasdfasdfasdfasdfasdfasdf" + +type 'a r = {mutable a: 'a; b: 'a; global_ c: 'a} + +type 'a r = + | Foo of global_ 'a + | Bar of 'a * global_ 'a + | Baz of global_ int * string * global_ 'a + +type ('a, 'b) cfn = + a:local_ 'a -> ?b:local_ b -> local_ 'a -> (int -> local_ 'b) + +let _ = local_ () + +let _ = exclave_ () + +let () = local_ x + +let () = exclave_ x + +let {b} = local_ () + +let {b} = exclave_ () + +let () = local_ r + +let () = exclave_ r + +let local_ x : string = "hi" +let (x : string) = local_ "hi" + +let (x : string) = exclave_ "hi" + +let x = local_ ("hi" : string) + +let x = exclave_ ("hi" : string) +let x : 'a . 'a -> 'a = local_ "hi" +let x : 'a . 'a -> 'a = exclave_ "hi" +let local_ f : 'a. 'a -> 'a = "hi" + +let foo () = + if true then (local_ ()); + () + +type loc_long_attrs = (string[@ocaml.local]) -> (string[@ocaml.local]) + +type loc_short_attrs = (string[@local]) -> (string[@local]) + +type global_long_attrs = + | Foo of { s : string[@ocaml.global] } + | Bar of (string[@ocaml.global]) + +type global_short_attrs = + | Foo of { s : string[@global] } + | Bar of (string[@global]) + +type global_short_attrs = + | Foo of { s : string[@global] } + | Bar of (string[@global]) + +let local_long_ext = [%ocaml.local] () + +let local_short_ext = [%local] () + +let exclave_long_ext = [%ocaml.exclave] () + +let exclave_short_ext = [%exclave] () + +let[@ocaml.local] upstream_local_attr_long x = x + +let[@ocaml.local never] upstream_local_attr_never_long x = x + +let[@ocaml.local always] upstream_local_attr_always_long x = x + +let[@ocaml.local maybe] upstream_local_attr_maybe_long x = x + +let[@local] upstream_local_attr_short x = x + +let[@local never] upstream_local_attr_never_short x = x + +let[@local always] upstream_local_attr_always_short x = x + +let[@local maybe] upstream_local_attr_maybe_short x = x + +let f x = (* a *) local_ + let y = 1 in + x + y + +let f x = (* a *) exclave_ + let y = 1 in + x + y + +let x = (* a *) local_ + let y = 1 in + y + +let x = (* a *) exclave_ + let y = 1 in + y diff --git a/test/passing/tests/local.ml.ref b/test/passing/tests/local.ml.ref new file mode 100644 index 0000000000..0e07f1f876 --- /dev/null +++ b/test/passing/tests/local.ml.ref @@ -0,0 +1,135 @@ +let f a b c = 1 + +let f (local_ a) ~foo:(local_ b) ?foo:(local_ c = 1) ~(local_ d) = () + +let f ~(local_ x) ~(local_ y : string) ?(local_ z : string) = () + +let xs = [(fun (local_ a) (type b) ~(local_ c) -> local_ 1)] + +let xs = [(fun (local_ a) (type b) ~(local_ c) -> exclave_ 1)] + +let f () = local_ + let a = [local_ 1] in + let local_ r = 1 in + let local_ f : 'a. 'a -> 'a = fun x -> local_ x in + let local_ g a b c : int = 1 in + let () = g (local_ fun () -> ()) in + local_ "asdfasdfasdfasdfasdfasdfasdf" + +let f () = exclave_ + let a = [exclave_ 1] in + let local_ r = 1 in + let local_ f : 'a. 'a -> 'a = fun x -> exclave_ x in + let local_ g a b c : int = 1 in + let () = g (exclave_ fun () -> ()) in + exclave_ "asdfasdfasdfasdfasdfasdfasdf" + +type 'a r = {mutable a: 'a; b: 'a; global_ c: 'a} + +type 'a r = + | Foo of global_ 'a + | Bar of 'a * global_ 'a + | Baz of global_ int * string * global_ 'a + +type ('a, 'b) cfn = + a:local_ 'a -> ?b:local_ b -> local_ 'a -> (int -> local_ 'b) + +let _ = local_ () + +let _ = exclave_ () + +let () = local_ x + +let () = exclave_ x + +let {b} = local_ () + +let {b} = exclave_ () + +let () = local_ r + +let () = exclave_ r + +let local_ x : string = "hi" + +let (x : string) = local_ "hi" + +let (x : string) = exclave_ "hi" + +let local_ x = ("hi" : string) + +let x = exclave_ ("hi" : string) + +let x : 'a. 'a -> 'a = local_ "hi" + +let x : 'a. 'a -> 'a = exclave_ "hi" + +let local_ f : 'a. 'a -> 'a = "hi" + +let foo () = + if true then (local_ ()) ; + () + +type loc_long_attrs = (string[@ocaml.local]) -> (string[@ocaml.local]) + +type loc_short_attrs = (string[@local]) -> (string[@local]) + +type global_long_attrs = + | Foo of {s: string [@ocaml.global]} + | Bar of (string[@ocaml.global]) + +type global_short_attrs = + | Foo of {s: string [@global]} + | Bar of (string[@global]) + +type global_short_attrs = + | Foo of {s: string [@global]} + | Bar of (string[@global]) + +let local_long_ext = [%ocaml.local] () + +let local_short_ext = [%local] () + +let exclave_long_ext = [%ocaml.exclave] () + +let exclave_short_ext = [%exclave] () + +let[@ocaml.local] upstream_local_attr_long x = x + +let[@ocaml.local never] upstream_local_attr_never_long x = x + +let[@ocaml.local always] upstream_local_attr_always_long x = x + +let[@ocaml.local maybe] upstream_local_attr_maybe_long x = x + +let[@local] upstream_local_attr_short x = x + +let[@local never] upstream_local_attr_never_short x = x + +let[@local always] upstream_local_attr_always_short x = x + +let[@local maybe] upstream_local_attr_maybe_short x = x + +let f x = + (* a *) + local_ + let y = 1 in + x + y + +let f x = + (* a *) + exclave_ + let y = 1 in + x + y + +let x = + (* a *) + local_ + let y = 1 in + y + +let x = + (* a *) + exclave_ + let y = 1 in + y diff --git a/test/passing/tests/module_type_strengthening.ml b/test/passing/tests/module_type_strengthening.ml new file mode 100644 index 0000000000..b4f2965413 --- /dev/null +++ b/test/passing/tests/module_type_strengthening.ml @@ -0,0 +1,17 @@ +module M : S with M = struct end + +module M : + S with Fooooooooooooooooooooooooooo(Foooooooooo.Foo)(Fooooooooooooo) + (Fooooooooooooo) = struct end + +module M : S (* foo *) with M = struct end + +module M : S with (* fooo *) M = struct end + +module rec M : S with M = struct end + +module type S = sig + module rec M : S with M +end + +module type S = (S with M [@foo]) diff --git a/test/passing/tests/ocp_indent_compat-break_colon_after.ml.ref b/test/passing/tests/ocp_indent_compat-break_colon_after.ml.ref index 5afc5d045f..d2b4749ff7 100644 --- a/test/passing/tests/ocp_indent_compat-break_colon_after.ml.ref +++ b/test/passing/tests/ocp_indent_compat-break_colon_after.ml.ref @@ -91,3 +91,8 @@ let add_edge target dep = Hashtbl.replace edge_count target (1 + try Hashtbl.find edge_count target with Not_found -> 0) ; if not (Hashtbl.mem edge_count dep) then Hashtbl.add edge_count dep 0 ) + +let iarray_fold_transf (f : numbering -> 'a -> numbering * 'b) n + (a : 'a iarray) : numbering * 'b iarray + = + match Iarray.length a with 0 -> (n, [::]) | 1 -> x diff --git a/test/passing/tests/ocp_indent_compat.ml b/test/passing/tests/ocp_indent_compat.ml index 224fa0e407..920ad76359 100644 --- a/test/passing/tests/ocp_indent_compat.ml +++ b/test/passing/tests/ocp_indent_compat.ml @@ -92,3 +92,9 @@ let add_edge target dep = Hashtbl.replace edge_count target (1 + try Hashtbl.find edge_count target with Not_found -> 0) ; if not (Hashtbl.mem edge_count dep) then Hashtbl.add edge_count dep 0 ) + +let iarray_fold_transf (f : numbering -> 'a -> numbering * 'b) n + (a : 'a iarray) + : numbering * 'b iarray + = + match Iarray.length a with 0 -> (n, [::]) | 1 -> x diff --git a/test/passing/tests/open-closing-on-separate-line.ml.ref b/test/passing/tests/open-closing-on-separate-line.ml.ref index 4ccac15195..4b7eecef9d 100644 --- a/test/passing/tests/open-closing-on-separate-line.ml.ref +++ b/test/passing/tests/open-closing-on-separate-line.ml.ref @@ -317,6 +317,16 @@ let _ = M.((* A *) [|f|] (* B *)) let _ = M.((* A *) [|f|] (* B *) [@warning "foo"] (* C *)) +let _ = M.([:f:] [@warning "foo"]) + +let _ = M.((* A *) [:f:]) + +let _ = M.([:f:] (* B *)) + +let _ = M.((* A *) [:f:] (* B *)) + +let _ = M.((* A *) [:f:] (* B *) [@warning "foo"] (* C *)) + let _ = M.([f] [@warning "foo"]) let _ = M.((* A *) [f]) diff --git a/test/passing/tests/open.ml b/test/passing/tests/open.ml index ca2faad072..23a1fe04c2 100644 --- a/test/passing/tests/open.ml +++ b/test/passing/tests/open.ml @@ -242,6 +242,16 @@ let _ = M.((* A *) [| f |] (* B *)) let _ = M.((* A *) [| f |] (* B *) [@warning "foo"] (* C *)) +let _ = M.([:f:] [@warning "foo"]) + +let _ = M.((* A *) [: f :]) + +let _ = M.([: f :] (* B *)) + +let _ = M.((* A *) [: f :] (* B *)) + +let _ = M.((* A *) [: f :] (* B *) [@warning "foo"] (* C *)) + let _ = M.([f] [@warning "foo"]) let _ = M.((* A *) [ f ]) diff --git a/test/passing/tests/open.ml.ref b/test/passing/tests/open.ml.ref index 26e0737169..535b7c554b 100644 --- a/test/passing/tests/open.ml.ref +++ b/test/passing/tests/open.ml.ref @@ -307,6 +307,16 @@ let _ = M.((* A *) [|f|] (* B *)) let _ = M.((* A *) [|f|] (* B *) [@warning "foo"] (* C *)) +let _ = M.([:f:] [@warning "foo"]) + +let _ = M.((* A *) [:f:]) + +let _ = M.([:f:] (* B *)) + +let _ = M.((* A *) [:f:] (* B *)) + +let _ = M.((* A *) [:f:] (* B *) [@warning "foo"] (* C *)) + let _ = M.([f] [@warning "foo"]) let _ = M.((* A *) [f]) diff --git a/test/passing/tests/override.ml.ref b/test/passing/tests/override.ml.ref index ea238082d3..b7fdbded42 100644 --- a/test/passing/tests/override.ml.ref +++ b/test/passing/tests/override.ml.ref @@ -1,6 +1,6 @@ let _ = {} -let _ = {} +let _ = {} let _ = {} diff --git a/test/passing/tests/polyparams.ml b/test/passing/tests/polyparams.ml new file mode 100644 index 0000000000..74ddf882c6 --- /dev/null +++ b/test/passing/tests/polyparams.ml @@ -0,0 +1,15 @@ +let poly1 (id : 'a. 'a -> 'a) = (id 3, id "three") + +let poly2 : ('a. 'a -> 'a) -> int * string = fun id -> (id 3, id "three") + +let poly3 : 'b. ('a. 'a -> 'a) -> 'b -> 'b * 'b option = + fun id x -> (id x, id (Some x)) + +let rec poly4 p (id : 'a. 'a -> 'a) = + if p then poly4 false id else (id 4, id "four") + +let rec poly5 : bool -> ('a. 'a -> 'a) -> int * string = + fun p id -> if p then poly5 false id else (id 5, id "five") + +let rec poly6 : 'b. bool -> ('a. 'a -> 'a) -> 'b -> 'b * 'b option = + fun p id x -> if p then poly6 false id x else (id x, id (Some x)) diff --git a/test/passing/tests/print_config.ml.err b/test/passing/tests/print_config.ml.err index 4557603aa5..f4b38a1ffa 100644 --- a/test/passing/tests/print_config.ml.err +++ b/test/passing/tests/print_config.ml.err @@ -7,6 +7,7 @@ ocaml-version=4.13.0 (file tests/.ocamlformat:7) quiet=false disable-conf-attrs=false version-check=true +rewrite-old-style-jane-street-local-annotations=false assignment-operator=end-line (profile ocamlformat (file tests/.ocamlformat:1)) break-before-in=fit-or-vertical (profile ocamlformat (file tests/.ocamlformat:1)) break-cases=fit (file tests/.ocamlformat:2) diff --git a/test/passing/tests/source.ml b/test/passing/tests/source.ml index 7d764f9e4d..31bc283523 100644 --- a/test/passing/tests/source.ml +++ b/test/passing/tests/source.ml @@ -7364,7 +7364,7 @@ type t = | ;; M.(Some x) [@foo] -[@@@foo:] +[@@@foo: ] let x = (A(B)).a diff --git a/test/passing/tests/source.ml.ref b/test/passing/tests/source.ml.ref index 0ec00cfffc..fba21cef75 100644 --- a/test/passing/tests/source.ml.ref +++ b/test/passing/tests/source.ml.ref @@ -2387,7 +2387,7 @@ let intAorB = function `TagA i -> i | `TagB -> 4 type _ wrapPoly = | WrapPoly : 'a poly -> ([< `TagA of int | `TagB] as 'a) wrapPoly -let example6 : type a. a wrapPoly -> a -> int = +let example6 : type a. a wrapPoly -> (a -> int) = fun w -> match w with | WrapPoly ATag -> intA @@ -9124,7 +9124,7 @@ type t = | ;; M.(Some x) [@foo] -[@@@foo:] +[@@@foo: ] let x = (A B).a diff --git a/test/passing/tests/unboxed_types.ml b/test/passing/tests/unboxed_types.ml new file mode 100644 index 0000000000..f3bb0a8ee9 --- /dev/null +++ b/test/passing/tests/unboxed_types.ml @@ -0,0 +1,41 @@ +let x = #3l + +let x = #3L + +let x = #3n + +let x = -#3l + +let x = #3.0 + +let x = -#3.0 + +let x = #3.0 + #4.0 + +let x = #3.0 - #4.0 + +let x = (#3.0 [@attr]) + +let x = (#3.0 + #4.0) [@attr] + +let x = f #3.0 #4.0 #5.0 x y #0. + +type t = float# + +type t = float# * float# + +type t = float# t2 + +type t = float #t2 + +type t = (int, float#) either + +type t = (float#, int) either + +type t = (float#, float#) either + +type ('a : float64) t = 'a + +type ('b, 'a : float64) t + +type ('b : float64, 'a : immediate) t diff --git a/test/passing/tests/unboxed_types2.ml b/test/passing/tests/unboxed_types2.ml new file mode 100644 index 0000000000..6727ce5455 --- /dev/null +++ b/test/passing/tests/unboxed_types2.ml @@ -0,0 +1,22 @@ +(* this one is purposefully misformatted *) + +let x = +#3.14 +let x = #3.14+#5.82 +let x = #3l-#4n +let x = #4n+ -#5.0 +let x = f #3n#4l#5.0-#6L+-#7.0z#3n +let x = f #3n#4l#5.0-#6L+-#7.0 z#m #3n + +type t = float#float +type t = float #float +type t = float# float +type t = float#float# +type t = #float float# #float + + +type ('a : float64) t = 'a +type ( 'b, 'a : float64) + t +type ('b : float64, 'a : immediate) t + = + 'a diff --git a/test/passing/tests/unboxed_types2.ml.ref b/test/passing/tests/unboxed_types2.ml.ref new file mode 100644 index 0000000000..0fd8283dcb --- /dev/null +++ b/test/passing/tests/unboxed_types2.ml.ref @@ -0,0 +1,29 @@ +(* this one is purposefully misformatted *) + +let x = #3.14 + +let x = #3.14 + #5.82 + +let x = #3l - #4n + +let x = #4n + -#5.0 + +let x = f #3n #4l #5.0 - #6L +- #7.0z #3n + +let x = f #3n #4l #5.0 - #6L +- #7.0 z#m #3n + +type t = float# float + +type t = float #float + +type t = float# float + +type t = float# float# + +type t = #float float# #float + +type ('a : float64) t = 'a + +type ('b, 'a : float64) t + +type ('b : float64, 'a : immediate) t = 'a diff --git a/test/passing/tests/verbose1.ml.err b/test/passing/tests/verbose1.ml.err index 7bf67de296..efd0f05c8b 100644 --- a/test/passing/tests/verbose1.ml.err +++ b/test/passing/tests/verbose1.ml.err @@ -7,6 +7,7 @@ ocaml-version=4.13.0 (file tests/.ocamlformat:7) quiet=false disable-conf-attrs=false version-check=true +rewrite-old-style-jane-street-local-annotations=false assignment-operator=end-line (profile ocamlformat (file tests/.ocamlformat:1)) break-before-in=fit-or-vertical (profile ocamlformat (file tests/.ocamlformat:1)) break-cases=fit (file tests/.ocamlformat:2) diff --git a/vendor/ocaml-common/location.ml b/vendor/ocaml-common/location.ml index a336e89dcd..9a1972e996 100644 --- a/vendor/ocaml-common/location.ml +++ b/vendor/ocaml-common/location.ml @@ -18,6 +18,36 @@ open Lexing type t = Warnings.loc = { loc_start: position; loc_end: position; loc_ghost: bool } +let equal + { loc_start = { pos_fname = loc_start_pos_fname_1 + ; pos_lnum = loc_start_pos_lnum_1 + ; pos_bol = loc_start_pos_bol_1 + ; pos_cnum = loc_start_pos_cnum_1 } + ; loc_end = { pos_fname = loc_end_pos_fname_1 + ; pos_lnum = loc_end_pos_lnum_1 + ; pos_bol = loc_end_pos_bol_1 + ; pos_cnum = loc_end_pos_cnum_1 } + ; loc_ghost = loc_ghost_1 } + { loc_start = { pos_fname = loc_start_pos_fname_2 + ; pos_lnum = loc_start_pos_lnum_2 + ; pos_bol = loc_start_pos_bol_2 + ; pos_cnum = loc_start_pos_cnum_2 } + ; loc_end = { pos_fname = loc_end_pos_fname_2 + ; pos_lnum = loc_end_pos_lnum_2 + ; pos_bol = loc_end_pos_bol_2 + ; pos_cnum = loc_end_pos_cnum_2 } + ; loc_ghost = loc_ghost_2 } + = + String.equal loc_start_pos_fname_1 loc_start_pos_fname_2 && + Int.equal loc_start_pos_lnum_1 loc_start_pos_lnum_2 && + Int.equal loc_start_pos_bol_1 loc_start_pos_bol_2 && + Int.equal loc_start_pos_cnum_1 loc_start_pos_cnum_2 && + String.equal loc_end_pos_fname_1 loc_end_pos_fname_2 && + Int.equal loc_end_pos_lnum_1 loc_end_pos_lnum_2 && + Int.equal loc_end_pos_bol_1 loc_end_pos_bol_2 && + Int.equal loc_end_pos_cnum_1 loc_end_pos_cnum_2 && + Bool.equal loc_ghost_1 loc_ghost_2 + let in_file = Warnings.ghost_loc_in_file let none = in_file "_none_" @@ -37,6 +67,11 @@ let init lexbuf fname = pos_cnum = 0; } +let ghostify l = + if l.loc_ghost + then l + else { l with loc_ghost = true } + let symbol_rloc () = { loc_start = Parsing.symbol_start_pos (); loc_end = Parsing.symbol_end_pos (); diff --git a/vendor/ocaml-common/location.mli b/vendor/ocaml-common/location.mli index 149447965a..19bd7c939a 100644 --- a/vendor/ocaml-common/location.mli +++ b/vendor/ocaml-common/location.mli @@ -35,6 +35,12 @@ type t = Warnings.loc = { Else all fields are correct. *) +(** Strict equality: Two locations are equal iff every field is equal. Two + locations that happen to refer to the same place -- for instance, if one has + [pos_lnum] set correctly and the other has [pos_lnum = -1] -- are not + considered to be equal. *) +val equal : t -> t -> bool + val none : t (** An arbitrary value of type [t]; describes an empty ghost range. *) @@ -51,6 +57,9 @@ val init : Lexing.lexbuf -> string -> unit val curr : Lexing.lexbuf -> t (** Get the location of the current token from the [lexbuf]. *) +val ghostify : t -> t +(** Return a version of the location with [loc_ghost = true] *) + val symbol_rloc: unit -> t val symbol_gloc: unit -> t diff --git a/vendor/parser-extended/ast_helper.ml b/vendor/parser-extended/ast_helper.ml index 442ce1b63f..2d8536bb5e 100644 --- a/vendor/parser-extended/ast_helper.ml +++ b/vendor/parser-extended/ast_helper.ml @@ -79,6 +79,10 @@ module Typ = struct let poly ?loc ?attrs a b = mk ?loc ?attrs (Ptyp_poly (a, b)) let package ?loc ?attrs a b = mk ?loc ?attrs (Ptyp_package (a, b)) let extension ?loc ?attrs a = mk ?loc ?attrs (Ptyp_extension a) + + (* Jane Street extension *) + let constr_unboxed ?loc ?attrs a b = mk ?loc ?attrs (Ptyp_constr_unboxed (a, b)) + (* End Jane Street extension *) end module Pat = struct @@ -197,6 +201,7 @@ module Mty = struct let with_ ?loc ?attrs a b = mk ?loc ?attrs (Pmty_with (a, b)) let typeof_ ?loc ?attrs a = mk ?loc ?attrs (Pmty_typeof a) let extension ?loc ?attrs a = mk ?loc ?attrs (Pmty_extension a) + let strengthen ?loc ?attrs a b = mk ?loc ?attrs (Pmty_strengthen (a, b)) end module Mod = struct @@ -599,3 +604,26 @@ module Of = struct let inherit_ ?loc ty = mk ?loc (Oinherit ty) end + +(* Jane Street extension *) +module Jane = struct + let sign_str = function + | Positive -> "" + | Negative -> "-" + + let pconst_unboxed_integer sign value suffix = + if Erase_jane_syntax.should_erase () + then Pconst_integer (sign_str sign ^ value, suffix) + else Pconst_unboxed_integer (sign, value, suffix) + + let pconst_unboxed_float sign value suffix = + if Erase_jane_syntax.should_erase () + then Pconst_float (sign_str sign ^ value, suffix) + else Pconst_unboxed_float (sign, value, suffix) + + let ptyp_constr_unboxed ident args = + if Erase_jane_syntax.should_erase () + then Ptyp_constr (ident, args) + else Ptyp_constr_unboxed (ident, args) +end +(* End Jane Street extension *) diff --git a/vendor/parser-extended/ast_mapper.ml b/vendor/parser-extended/ast_mapper.ml index b1da740668..515e5c3c91 100644 --- a/vendor/parser-extended/ast_mapper.ml +++ b/vendor/parser-extended/ast_mapper.ml @@ -147,6 +147,12 @@ module C = struct pconst_desc | Pconst_string (s, loc, quotation_delimiter) -> Pconst_string (s, sub.location sub loc, quotation_delimiter) + + (* Jane Street extension *) + | Pconst_unboxed_integer _ + | Pconst_unboxed_float _ + -> pconst_desc + (* End Jane Street extension *) in Const.mk ~loc desc end @@ -215,6 +221,11 @@ module T = struct package ~loc ~attrs lid l | Ptyp_extension x -> extension ~loc ~attrs (sub.extension sub x) + (* Jane Street extension *) + | Ptyp_constr_unboxed (lid, tl) -> + constr_unboxed ~loc ~attrs (map_loc sub lid) (List.map (sub.typ sub) tl) + (* End Jane Street extension *) + let map_type_declaration sub {ptype_name; ptype_params; ptype_cstrs; ptype_kind; @@ -364,6 +375,10 @@ module MT = struct (List.map (sub.with_constraint sub) l) | Pmty_typeof me -> typeof_ ~loc ~attrs (sub.module_expr sub me) | Pmty_extension x -> extension ~loc ~attrs (sub.extension sub x) + | Pmty_strengthen (mt,s) -> + strengthen ~loc ~attrs + (sub.module_type sub mt) + (map_loc sub s) let map_with_constraint sub = function | Pwith_type (lid, d) -> diff --git a/vendor/parser-extended/asttypes.mli b/vendor/parser-extended/asttypes.mli index a277629af1..806b83876d 100644 --- a/vendor/parser-extended/asttypes.mli +++ b/vendor/parser-extended/asttypes.mli @@ -52,6 +52,19 @@ type obj_closed_flag = | OClosed | OOpen of Location.t +type global_flag = + | Global + | Nothing + +(* constant layouts are parsed as layout annotations, and also used + in the type checker as already-inferred (i.e. non-variable) layouts *) +type const_layout = + | Any + | Value + | Void + | Immediate64 + | Immediate + type label = string type 'a loc = 'a Location.loc = { diff --git a/vendor/parser-extended/dune b/vendor/parser-extended/dune index 72633eb613..448c9e1cad 100644 --- a/vendor/parser-extended/dune +++ b/vendor/parser-extended/dune @@ -3,7 +3,12 @@ (public_name ocamlformat-lib.parser_extended) (flags (:standard -w -9 -open Parser_shims -open Ocaml_common)) - (libraries compiler-libs.common menhirLib parser_shims ocaml_common)) + (libraries + erase_jane_syntax + compiler-libs.common + menhirLib + parser_shims + ocaml_common)) (ocamllex lexer) diff --git a/vendor/parser-extended/extensions.ml b/vendor/parser-extended/extensions.ml new file mode 100644 index 0000000000..b24277acd0 --- /dev/null +++ b/vendor/parser-extended/extensions.ml @@ -0,0 +1,354 @@ +open Asttypes +open Parsetree +open Extensions_parsing + +(******************************************************************************) +(** Individual language extension modules *) + +(* Note [Check for immutable extension in comprehensions code] + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + + When we spot a comprehension for an immutable array, we need to make sure + that both [comprehensions] and [immutable_arrays] are enabled. But our + general mechanism for checking for enabled extensions (in + Extensions_parsing.Translate(...).of_ast) won't work well here: it triggers + when converting from e.g. [[%extensions.comprehensions.array] ...] to the + comprehensions-specific AST. But if we spot a + [[%extensions.comprehensions.immutable]], there is no expression to + translate. So we just check for the immutable arrays extension when + processing a comprehension expression for an immutable array. + + Note [Wrapping with make_extension] + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + + The topmost node in the encoded AST must always look like e.g. + [%extension.comprehensions]. This allows the decoding machinery to know + what extension is being used and what function to call to do the decoding. + Accordingly, during encoding, after doing the hard work of converting the + extension syntax tree into e.g. Parsetree.expression, we need to make a final + step of wrapping the result in an [%extension.xyz] node. Ideally, this step + would be done by part of our general structure, like we separate [of_ast] + and [of_ast_internal] in the decode structure; this design would make it + structurally impossible/hard to forget taking this final step. + + However, the final step is only one line of code (a call to + [make_extension]), but yet the name of the extension varies, as does the type + of the payload. It would thus take several lines of code to execute this + command otherwise, along with dozens of lines to create the structure in the + first place. And so instead we just manually call [make_extension] and refer + to this Note as a reminder to authors of future extensions to remember to do + this wrapping. +*) + +(** List and array comprehensions *) +module Comprehensions = struct + let extension_string = Language_extension.to_string Comprehensions + + type iterator = + | Range of { start : expression + ; stop : expression + ; direction : direction_flag } + | In of expression + + type clause_binding = + { pattern : pattern + ; iterator : iterator + ; attributes : attribute list } + + type clause = + | For of clause_binding list + | When of expression + + type comprehension = + { body : expression + ; clauses : clause list + } + + type expression = + | Cexp_list_comprehension of comprehension + | Cexp_array_comprehension of mutable_flag * comprehension + + (* The desugared-to-OCaml version of comprehensions is described by the + following BNF, where [{% '...' | expr %}] refers to the result of + [Expression.make_extension] (via [comprehension_expr]) as described at the + top of [extensions_parsing.mli]. + + {v + comprehension ::= + | {% 'comprehension.list' | '[' clauses ']' %} + | {% 'comprehension.array' | '[|' clauses '|]' %} + + clauses ::= + | {% 'comprehension.for' | 'let' iterator+ 'in' clauses %} + | {% 'comprehension.when' | expr ';' clauses %} + | {% 'comprehension.body' | expr %} + + iterator ::= + | pattern '=' {% 'comprehension.for.range.upto' | expr ',' expr %} + | pattern '=' {% 'comprehension.for.range.downto' | expr ',' expr %} + | pattern '=' {% 'comprehension.for.in' | expr %} + v} + *) + + let comprehension_expr ~loc names = + Expression.make_extension ~loc (extension_string :: names) + + (** First, we define how to go from the nice AST to the OCaml AST; this is + the [expr_of_...] family of expressions, culminating in + [expr_of_comprehension_expr]. *) + + let expr_of_iterator ~loc = function + | Range { start; stop; direction } -> + comprehension_expr + ~loc + [ "for" + ; "range" + ; match direction with + | Upto -> "upto" + | Downto -> "downto" ] + (Ast_helper.Exp.tuple [start; stop]) + | In seq -> + comprehension_expr ~loc ["for"; "in"] seq + + let expr_of_clause_binding ~loc { pattern; iterator; attributes } = + Ast_helper.Vb.mk + ~loc + ~attrs:attributes + ~is_pun:false + pattern + (expr_of_iterator ~loc iterator) + + let expr_of_clause ~loc clause rest = match clause with + | For iterators -> + comprehension_expr + ~loc + ["for"] + (Ast_helper.Exp.let_ + { pvbs_bindings = + List.map (expr_of_clause_binding ~loc) iterators; + pvbs_rec = Nonrecursive; + pvbs_extension = None } + rest) + | When cond -> + comprehension_expr + ~loc + ["when"] + (Ast_helper.Exp.sequence cond rest) + + let expr_of_comprehension ~loc ~type_ { body; clauses } = + comprehension_expr + ~loc + type_ + (List.fold_right + (expr_of_clause ~loc) + clauses + (comprehension_expr ~loc ["body"] body)) + + let expr_of ~loc eexpr = + let ghost_loc = { loc with Location.loc_ghost = true } in + let expr_of_comprehension_type type_ = + expr_of_comprehension ~loc:ghost_loc ~type_ + in + (* See Note [Wrapping with make_extension] *) + Expression.make_extension ~loc [extension_string] @@ + match eexpr with + | Cexp_list_comprehension comp -> + expr_of_comprehension_type ["list"] comp + | Cexp_array_comprehension (amut, comp) -> + expr_of_comprehension_type + [ "array" + ; match amut with + | Mutable _ -> + "mutable" + | Immutable -> + "immutable" + ] + comp + + (** Then, we define how to go from the OCaml AST to the nice AST; this is + the [..._of_expr] family of expressions, culminating in + [comprehension_expr_of_expr]. *) + + module Desugaring_error = struct + type error = + | Non_comprehension_extension_point of string list + | Non_extension + | Bad_comprehension_extension_point of string list + | No_clauses + + let report_error ~loc = function + | Non_comprehension_extension_point name -> + Location.errorf ~loc + "Tried to desugar the non-comprehension extension point \ + \"extension.%s\" as part of a comprehension expression" + (String.concat "." name) + | Non_extension -> + Location.errorf ~loc + "Tried to desugar a non-extension expression as part of a \ + comprehension expression" + | Bad_comprehension_extension_point name -> + Location.errorf ~loc + "Unknown, unexpected, or malformed comprehension extension point \ + \"extension.comprehension.%s\"" + (String.concat "." name) + | No_clauses -> + Location.errorf ~loc + "Tried to desugar a comprehension with no clauses" + + exception Error of Location.t * error + + let () = + Location.register_error_of_exn + (function + | Error(loc, err) -> Some (report_error ~loc err) + | _ -> None) + + let raise expr err = raise (Error(expr.pexp_loc, err)) + end + + let expand_comprehension_extension_expr expr = + match Expression.match_extension expr with + | Some (comprehensions :: name, expr) + when String.equal comprehensions extension_string -> + name, expr + | Some (name, _) -> + Desugaring_error.raise expr (Non_comprehension_extension_point name) + | None -> + Desugaring_error.raise expr Non_extension + + let iterator_of_expr expr = + match expand_comprehension_extension_expr expr with + | ["for"; "range"; "upto"], + { pexp_desc = Pexp_tuple [start; stop]; _ } -> + Range { start; stop; direction = Upto } + | ["for"; "range"; "downto"], + { pexp_desc = Pexp_tuple [start; stop]; _ } -> + Range { start; stop; direction = Downto } + | ["for"; "in"], seq -> + In seq + | bad, _ -> + Desugaring_error.raise expr (Bad_comprehension_extension_point bad) + + let clause_binding_of_pvb { pvb_pat; pvb_expr; pvb_attributes; _ } = + { pattern = pvb_pat + ; iterator = iterator_of_expr pvb_expr + ; attributes = pvb_attributes } + + let add_clause clause comp = { comp with clauses = clause :: comp.clauses } + + let rec raw_comprehension_of_expr expr = + match expand_comprehension_extension_expr expr with + | ["for"], { pexp_desc = Pexp_let({ pvbs_rec = Nonrecursive; + pvbs_bindings = iterators }, + rest); _ } -> + add_clause + (For (List.map clause_binding_of_pvb iterators)) + (raw_comprehension_of_expr rest) + | ["when"], { pexp_desc = Pexp_sequence(cond, rest); _ } -> + add_clause + (When cond) + (raw_comprehension_of_expr rest) + | ["body"], body -> + { body; clauses = [] } + | bad, _ -> + Desugaring_error.raise expr (Bad_comprehension_extension_point bad) + + let comprehension_of_expr expr = + match raw_comprehension_of_expr expr with + | { body = _; clauses = [] } -> + Desugaring_error.raise expr No_clauses + | comp -> comp + + let comprehension_expr_of_expr expr = + match expand_comprehension_extension_expr expr with + | ["list"], comp -> + Cexp_list_comprehension (comprehension_of_expr comp) + | ["array"; "mutable"], comp -> + Cexp_array_comprehension (Mutable expr.pexp_loc, + comprehension_of_expr comp) + | ["array"; "immutable"], comp -> + (* assert_extension_enabled: + See Note [Check for immutable extension in comprehensions code] *) + assert_extension_enabled ~loc:expr.pexp_loc Immutable_arrays; + Cexp_array_comprehension (Immutable, comprehension_of_expr comp) + | bad, _ -> + Desugaring_error.raise expr (Bad_comprehension_extension_point bad) +end + +(** Immutable arrays *) +module Immutable_arrays = struct + type nonrec expression = + | Iaexp_immutable_array of expression list + + type nonrec pattern = + | Iapat_immutable_array of pattern list + + let extension_string = Language_extension.to_string Immutable_arrays + + let expr_of ~loc = function + | Iaexp_immutable_array elts -> + (* See Note [Wrapping with make_extension] *) + Expression.make_extension ~loc [extension_string] @@ + Ast_helper.Exp.array ~loc elts + + let of_expr expr = match expr.pexp_desc with + | Pexp_array elts -> Iaexp_immutable_array elts + | _ -> failwith "Malformed immutable array expression" + + let pat_of ~loc = function + | Iapat_immutable_array elts -> + (* See Note [Wrapping with make_extension] *) + Pattern.make_extension ~loc [extension_string] @@ + Ast_helper.Pat.array ~loc elts + + let of_pat expr = match expr.ppat_desc with + | Ppat_array elts -> Iapat_immutable_array elts + | _ -> failwith "Malformed immutable array expression" +end + +(******************************************************************************) +(** The interface to language extensions, which we export *) + +module type AST = sig + type t + type ast + + val of_ast : ast -> t option +end + +module Expression = struct + module M = struct + module AST = Extensions_parsing.Expression + + type t = + | Eexp_comprehension of Comprehensions.expression + | Eexp_immutable_array of Immutable_arrays.expression + + let of_ast_internal (ext : Language_extension.t) expr = match ext with + | Comprehensions -> + Some (Eexp_comprehension (Comprehensions.comprehension_expr_of_expr expr)) + | Immutable_arrays -> + Some (Eexp_immutable_array (Immutable_arrays.of_expr expr)) + | _ -> None + end + + include M + include Make_of_ast(M) +end + +module Pattern = struct + module M = struct + module AST = Extensions_parsing.Pattern + + type t = + | Epat_immutable_array of Immutable_arrays.pattern + + let of_ast_internal (ext : Language_extension.t) pat = match ext with + | Immutable_arrays -> + Some (Epat_immutable_array (Immutable_arrays.of_pat pat)) + | _ -> None + end + + include M + include Make_of_ast(M) +end diff --git a/vendor/parser-extended/extensions.mli b/vendor/parser-extended/extensions.mli new file mode 100644 index 0000000000..683a3510a1 --- /dev/null +++ b/vendor/parser-extended/extensions.mli @@ -0,0 +1,145 @@ +(** Syntax for our custom ocaml-jst language extensions. This module provides + two things: + + 1. First-class ASTs for all syntax introduced by our language extensions, + one for each OCaml AST we extend, divided up into one extension per + module and all available at once through modules named after the + syntactic category ([Expression.t], etc.). + + 2. A way to interpret these values as terms of the coresponding OCaml ASTs, + and to match on terms of those OCaml ASTs to see if they're language + extension terms. + + We keep our language extensions separate so that we can avoid having to + modify the existing AST, as this would break compatibility with every + existing ppx. + + For details on the rationale behind this approach (and for some of the gory + details), see [Extensions_parsing]. *) + +(** The ASTs for list and array comprehensions *) +module Comprehensions : sig + type iterator = + | Range of { start : Parsetree.expression + ; stop : Parsetree.expression + ; direction : Asttypes.direction_flag } + (** "= START to STOP" (direction = Upto) + "= START downto STOP" (direction = Downto) *) + | In of Parsetree.expression + (** "in EXPR" *) + + (* In [Typedtree], the [pattern] moves into the [iterator]. *) + type clause_binding = + { pattern : Parsetree.pattern + ; iterator : iterator + ; attributes : Parsetree.attribute list } + (** PAT (in/=) ... [@...] *) + + type clause = + | For of clause_binding list + (** "for PAT (in/=) ... and PAT (in/=) ... and ..."; must be nonempty *) + | When of Parsetree.expression + (** "when EXPR" *) + + type comprehension = + { body : Parsetree.expression + (** The body/generator of the comprehension *) + ; clauses : clause list + (** The clauses of the comprehension; must be nonempty *) } + + type expression = + | Cexp_list_comprehension of comprehension + (** [BODY ...CLAUSES...] *) + | Cexp_array_comprehension of Asttypes.mutable_flag * comprehension + (** [|BODY ...CLAUSES...|] (flag = Mutable) + [:BODY ...CLAUSES...:] (flag = Immutable) + (only allowed with [-extension immutable_arrays]) *) + + val expr_of : loc:Location.t -> expression -> Parsetree.expression +end + +(** The ASTs for immutable arrays. When we merge this upstream, we'll merge + these into the existing [P{exp,pat}_array] constructors by adding a + [mutable_flag] argument (just as we did with [T{exp,pat}_array]). *) +module Immutable_arrays : sig + type expression = + | Iaexp_immutable_array of Parsetree.expression list + (** [: E1; ...; En :] *) + + type pattern = + | Iapat_immutable_array of Parsetree.pattern list + (** [: P1; ...; Pn :] **) + + val expr_of : loc:Location.t -> expression -> Parsetree.expression + val pat_of : loc:Location.t -> pattern -> Parsetree.pattern +end + +(** The module type of language extension ASTs, instantiated once for each + syntactic category. We tend to call the pattern-matching functions here + with unusual indentation, not indenting the [None] branch further so as to + avoid merge conflicts with upstream. *) +module type AST = sig + (** The AST for all our ocaml-jst language extensions; one constructor per + language extension that extends the expression language. Some extensions + are handled separately and thus are not listed here. *) + type t + + (** The corresponding OCaml AST *) + type ast + + (** Given an OCaml AST node, check to see if it corresponds to a language + extension term. If it is, and the extension is enabled, then return it; + if it's not a language extension term, return [None]; if it's a disabled + language extension term, raise an error. + + AN IMPORTANT NOTE: We indent calls to this function *very* strangely: we + *do not change the indentation level* when we match on its result! + E.g. from [type_expect_] in [typecore.ml]: + + {[ + match Extensions.Expression.of_ast sexp with + | Some eexp -> + type_expect_extension + ~loc ~env ~expected_mode ~ty_expected ~explanation eexp + | None -> match sexp.pexp_desc with + | Pexp_ident lid -> + let path, mode, desc, kind = type_ident env ~recarg lid in + (* ... *) + | Pexp_constant(Pconst_string (str, _, _) as cst) -> + register_allocation expected_mode; + (* ... *) + | (* ... *) + | Pexp_unreachable -> + re { exp_desc = Texp_unreachable; + exp_loc = loc; exp_extra = []; + exp_type = instance ty_expected; + exp_mode = expected_mode.mode; + exp_attributes = sexp.pexp_attributes; + exp_env = env } + ]} + + Note that we match on the result of this function, forward to + [type_expect_extension] if we get something, and otherwise do the real + match on [sexp.pexp_desc] *without going up an indentation level*. This + is important to reduce the number of merge conflicts with upstream by + avoiding changing the body of every single important function in the type + checker to add pointless indentation. *) + val of_ast : ast -> t option +end + +(** Language extensions in expressions *) +module Expression : sig + type t = + | Eexp_comprehension of Comprehensions.expression + | Eexp_immutable_array of Immutable_arrays.expression + + include AST with type t := t and type ast := Parsetree.expression +end + +(** Language extensions in patterns *) +module Pattern : sig + type t = + | Epat_immutable_array of Immutable_arrays.pattern + + include AST with type t := t and type ast := Parsetree.pattern +end diff --git a/vendor/parser-extended/extensions_parsing.ml b/vendor/parser-extended/extensions_parsing.ml new file mode 100644 index 0000000000..824719e230 --- /dev/null +++ b/vendor/parser-extended/extensions_parsing.ml @@ -0,0 +1,338 @@ +(** As mentioned in the .mli file, there are some gory details around the + particular translation scheme we adopt for moving to and from OCaml ASTs + ([Parsetree.expression], etc.). The general idea is that we adopt a scheme + where a language extension is represented as a pair of an extension node and + an AST item that serves as the "body". In particular, for an extension + named [EXTNAME] (i.e., one that is enabled by [-extension EXTNAME] on the + command line), the extension node used must be [[%extension.EXTNAME]]. We + also provide utilities for further desugaring similar applications where the + extension nodes have the longer form [[%extension.EXTNAME.ID1.ID2.….IDn]] + (with the outermost one being the [n = 0] case), as these might be used + inside the [EXPR]. (For example, within the outermost + [[%extension.comprehensions]] term for list and array comprehensions, we can + also use [[%extension.comprehensions.list]], + [[%extension.comprehensions.array]], [[%extensions.comprehensions.for.in]], + etc.). + + As mentioned, we represent terms as a "pair" and don't use the extension + node payload; this is so that ppxen can see inside these extension nodes. + If we put the subexpressions inside the extension node payload, then we + couldn't write something like [[[%string "Hello, %{x}!"] for x in names]], + as [ppx_string] wouldn't traverse inside the payload to find the [[%string]] + extension point. + + Language extensions are of course allowed to impose extra constraints + constraints on what legal bodies are; we're also happy for this translation + to error in various ways on malformed input, since nobody should ever be + writing these forms directly. They're just an implementation detail. + + See modules of type AST below to see how different syntactic categories + are represented. For example, expressions are rendered as an application + of the extension node to the body, i.e. [([%extension.EXTNAME] EXPR)]. + + We provide one module per syntactic category (e.g., [Expression]), of module + type [AST]. They also provide some simple machinery for working with the + general [%extension.EXTNAME.ID1.ID2.….IDn] wrapped forms. To construct + one, we provide [extension_expr]; to destructure one, we provide + [match_extension] in the various AST modules; to construct one, we provide + [make_extension] in the same places.. We still have to write the + transformations in both directions for all new syntax, lowering it to + extension nodes and then lifting it back out. *) + +open Parsetree + +(******************************************************************************) +module Error = struct + type malformed_extension = + | Has_payload of payload + | Wrong_arguments of (Asttypes.arg_label * expression) list + | Wrong_tuple of pattern list + + type error = + | Malformed_extension of string list * malformed_extension + | Unknown_extension of string + | Disabled_extension of Language_extension.t + | Wrong_syntactic_category of Language_extension.t * string + | Unnamed_extension + | Bad_introduction of string * string list + + exception Error of Location.t * error +end + +open Error + +let assert_extension_enabled ~loc ext = + if not (Language_extension.is_enabled ext) then + raise (Error(loc, Disabled_extension ext)) +;; + +let report_error ~loc = function + | Malformed_extension(name, malformed) -> begin + let name = String.concat "." ("extension" :: name) in + match malformed with + | Has_payload _payload -> + Location.errorf + ~loc + "@[Modular extension nodes are not allowed to have a payload,@ \ + but \"%s\" does@]" + name + | Wrong_arguments arguments -> + Location.errorf + ~loc + "@[Expression modular extension nodes must be applied to exactly@ \ + one unlabeled argument, but \"%s\" was applied to@ %s@]" + name + (match arguments with + | [Labelled _, _] -> "a labeled argument" + | [Optional _, _] -> "an optional argument" + | _ -> Int.to_string (List.length arguments) ^ " arguments") + | Wrong_tuple patterns -> + Location.errorf + ~loc + "@[Pattern modular extension nodes must be the first component of@ \ + a pair, but \"%s\" was the first component of a %d-tuple@]" + name + (1 + List.length patterns) + end + | Unknown_extension name -> + Location.errorf + ~loc + "@[Unknown extension \"%s\" referenced via an@ [%%extension.%s] \ + extension node@]" + name + name + | Disabled_extension ext -> + Location.errorf + ~loc + "The extension \"%s\" is disabled and cannot be used" + (Language_extension.to_string ext) + | Wrong_syntactic_category(ext, cat) -> + Location.errorf + ~loc + "The extension \"%s\" cannot appear in %s" + (Language_extension.to_string ext) + cat + | Unnamed_extension -> + Location.errorf + ~loc + "Cannot have an extension node named [%%extension]" + | Bad_introduction(name, subnames) -> + Location.errorf + ~loc + "@[The extension \"%s\" was referenced improperly; it started with an@ \ + [%%extension.%s] extension node,@ not an [%%extension.%s] one@]" + name + (String.concat "." (name :: subnames)) + name + +let () = + Location.register_error_of_exn + (function + | Error(loc, err) -> Some (report_error ~loc err) + | _ -> None) + +(******************************************************************************) +(** Generically find and create the OCaml AST syntax used to encode one of our + language extensions. One module per variety of AST (expressions, patterns, + etc.). *) + +(** The parameters that define how to look for [[%extension.EXTNAME]] inside + ASTs of a certain syntactic category. See also the [Make_AST] functor, which + uses these definitions to make the e.g. [Expression] module. *) +module type AST_parameters = sig + (** The AST type (e.g., [Parsetree.expression]) *) + type ast + + (** The type of the subterms that occur in the "body" slot of an extension + use. This may just be [ast], but e.g. for expressions, we use function + applications, and the terms that a function is applied to contain label + information. *) + type raw_body + + (** The name for this syntactic category in the plural form; used for error + messages *) + val plural : string + + (** How to get the location attached to an AST node. Should just be + [fun tm -> tm.pCAT_loc] for the appropriate syntactic category [CAT]. *) + val location : ast -> Location.t + + (** How to construct an extension node for this AST (something of the shape + [[%name]] or [[%%name]], depending on the AST). Should just be + [Ast_helper.CAT.extension] for the appropriate syntactic category + [CAT]. *) + val make_extension_node : + ?loc:Location.t -> ?attrs:attributes -> extension -> ast + + (** Given an extension node (as created by [make_extension_node]) with an + appropriately-formed name and a body, combine them into the special + syntactic form we use for language extensions for this syntactic + category. Partial inverse of [match_extension_use]. *) + val make_extension_use : loc:Location.t -> extension_node:ast -> ast -> ast + + (** Given an AST node, check if it's of the special syntactic form indicating + that this is a language extension (as created by [make_extension_node]), + split it back up into the extension node and the possible body terms. + Doesn't do any checking about the name/format of the extension or the + possible body terms (see [AST.match_extension]). Partial inverse of + [make_extension_use]. *) + val match_extension_use : ast -> (extension * raw_body list) option + + (** Check if a [raw_body] term is legal to use as a body *) + val validate_extension_body : raw_body -> ast option + + (** The error to throw when the list of possible body terms is wrong: either + when the list isn't exactly one term long, or when that single term fails + [validate_extension_body] *) + val malformed_extension : raw_body list -> malformed_extension +end + +module type AST = sig + type ast + + val plural : string + + val location : ast -> Location.t + + val make_extension : loc:Location.t -> string list -> ast -> ast + + val match_extension : ast -> (string list * ast) option +end + +(* Some extensions written before this file existed are handled in their own + way; this function filters them out. *) +let uniformly_handled_extension names = + match names with + | [("local"|"global"|"nonlocal"|"escape"|"include_functor"|"curry"|"exclave")] -> false + | _ -> true + +(** Given the [AST_parameters] for a syntactic category, produce the + corresponding module, of type [AST], for lowering and lifting language + extension syntax from and to it. *) +module Make_AST (AST_parameters : AST_parameters) : + AST with type ast = AST_parameters.ast = struct + include AST_parameters + + let make_extension ~loc names = + make_extension_use + ~loc + ~extension_node: + (make_extension_node + ~loc + ({ txt = String.concat "." ("extension" :: names); loc }, PStr [])) + + (* This raises an error if the language extension node is malformed. Malformed + means either: + + 1. The [[%extension.NAME]] extension point has a payload; extensions must + be empty, so other ppxes can traverse "into" them. + + 2. The [[%extension.NAME]] extension point contains multiple body forms, or + body forms that are "shaped" incorrectly. *) + let match_extension ast = + match match_extension_use ast with + | Some (({txt = ext_name; loc = ext_loc}, ext_payload), body_list) -> begin + match String.split_on_char '.' ext_name with + | "extension" :: names when uniformly_handled_extension names -> begin + let raise_malformed err = + raise (Error(ext_loc, Malformed_extension(names, err))) + in + match ext_payload with + | PStr [] -> begin + match List.map validate_extension_body body_list with + | [Some body] -> Some (names, body) + | _ -> raise_malformed (malformed_extension body_list) + end + | _ -> raise_malformed (Has_payload ext_payload) + end + | _ -> None + end + | None -> None +end + +(** Expressions; embedded as [([%extension.EXTNAME] BODY)]. *) +module Expression = Make_AST(struct + type ast = expression + type raw_body = Asttypes.arg_label * expression (* Function arguments *) + + let plural = "expressions" + + let location expr = expr.pexp_loc + + let make_extension_node = Ast_helper.Exp.extension + + let make_extension_use ~loc ~extension_node expr = + Ast_helper.Exp.apply ~loc extension_node [Nolabel, expr] + + let match_extension_use expr = + match expr.pexp_desc with + | Pexp_apply({pexp_desc = Pexp_extension ext; _}, arguments) -> + Some (ext, arguments) + | _ -> + None + + let validate_extension_body = function + | Asttypes.Nolabel, body -> Some body + | _, _ -> None + + let malformed_extension args = Wrong_arguments args +end) + +(** Patterns; embedded as [[%extension.EXTNAME], BODY]. *) +module Pattern = Make_AST(struct + type ast = pattern + type raw_body = pattern + + let plural = "patterns" + + let location pat = pat.ppat_loc + + let make_extension_node = Ast_helper.Pat.extension + + let make_extension_use ~loc ~extension_node pat = + Ast_helper.Pat.tuple ~loc [extension_node; pat] + + let match_extension_use pat = + match pat.ppat_desc with + | Ppat_tuple({ppat_desc = Ppat_extension ext; _} :: patterns) -> + Some (ext, patterns) + | _ -> + None + + let validate_extension_body = Option.some + let malformed_extension pats = Wrong_tuple pats +end) + +(******************************************************************************) +(** Generically lift and lower our custom language extension ASTs from/to OCaml + ASTs. *) + +module type Of_ast_parameters = sig + module AST : AST + type t + val of_ast_internal : Language_extension.t -> AST.ast -> t option +end + +module Make_of_ast (Params : Of_ast_parameters) : sig + val of_ast : Params.AST.ast -> Params.t option +end = struct + let of_ast ast = + let loc = Params.AST.location ast in + let raise_error err = raise (Error (loc, err)) in + match Params.AST.match_extension ast with + | None -> None + | Some ([name], ast) -> begin + match Language_extension.of_string name with + | Some ext -> begin + assert_extension_enabled ~loc ext; + match Params.of_ast_internal ext ast with + | Some ext_ast -> Some ext_ast + | None -> raise_error (Wrong_syntactic_category(ext, Params.AST.plural)) + end + | None -> raise_error (Unknown_extension name) + end + | Some ([], _) -> + raise_error Unnamed_extension + | Some (name :: subnames, _) -> + raise_error (Bad_introduction(name, subnames)) +end diff --git a/vendor/parser-extended/extensions_parsing.mli b/vendor/parser-extended/extensions_parsing.mli new file mode 100644 index 0000000000..faf19c95e3 --- /dev/null +++ b/vendor/parser-extended/extensions_parsing.mli @@ -0,0 +1,182 @@ +(** This module handles the logic around the syntax of our extensions to OCaml + for [ocaml-jst], keeping the gory details wrapped up behind a clean + interface. + + As we've started to work on syntactic extensions to OCaml, three concerns + arose about the mechanics of how we wanted to maintain these changes in our + fork. + + 1. We don't want to extend the AST for our fork, as we really want to make + sure things like ppxen are cross-compatible between upstream and + [ocaml-jst]. Thankfully, OCaml already provides places to add extra + syntax: extension points and annotations! Thus, we have to come up with + a way of representing our new syntactic constructs in terms of extension + points (or annotations, but we went with the former). + + 2. We don't want to actually match on extension points whose names are + specific strings all over the compiler; that's incredibly messy, and it's + easy to miss cases, etc. + + 3. We want to keep different language extensions distinct so that we can add + them to upstream independently, work on them separately, and so on. + + We have come up with a design that addresses those concerns by providing + both a nice compiler-level interface for working with our syntactic + extensions as first-class AST nodes, as well as a uniform scheme for + translating this to and from OCaml AST values containing extension points. + One wrinkle is that OCaml has many ASTs, one for each syntactic category + (expressions, patterns, etc.); we have to provide this facility for each + syntactic category where we want to provide extensions. + + a. For each language extension, we will define a module (e.g., + [Comprehensions]), in which we define a proper AST type per syntactic + category we care about (e.g., [Comprehensions.expression] and its + subcomponents). This addresses concern (3); we've now contained each + extension in a module. But just that would leave them too siloed, so… + + b. We define an *overall auxiliary AST* for each syntactic category that's + just for our language extensions; for expressions, it's called + [Extensions.Expression.t]. It contains one constructor for each of the AST types + defined as described in design point (1). This addresses concern (2); we + can now match on actual OCaml constructors, as long as we can get a hold + of them. And to do that… + + c. We define a general scheme for how we represent language extensions in terms + of the existing ASTs, and provide a few primitives for consuming/creating + AST nodes of this form, for each syntactic category. There's not a lot + of abstraction to be done, or at least it's not (yet) apparent what + abstraction there is to do, so most of this remains manual. (Setting up + a full lens-based/otherwise bidirectional approach sounds like a great + opportunity for yak-shaving, but not *actually* a good idea.) This + solves concern (3), and by doing it uniformly helps us address multiple + cases at one stroke. + + Then, for each syntactic category, we define a module (in extensions.ml) + that contains functions for converting between the Parsetree representation + and the extension representation. A little functor magic (see [Make_of_ast]) + then allows us to make nice functions for export. + + This module contains the logic for moving to and from OCaml ASTs; the gory + details of the encoding are detailed in the implementation. All the actual + ASTs should live in [Extensions], which is the only module that should + directly depend on this one. + + When using this module, we often want to specify what our syntax extensions + look like when desugared into OCaml ASTs, so that we can validate the + translation code. We generally specify this as a BNF grammar, but we don't + want to depend on the specific details of the desugaring. Thus, instead of + writing out extension points or attributes directly, we write the result of + [Some_ast.make_extension ~loc [name1; name2; ...; NameN] a] as the special + syntax [{% 'name1.name2.....nameN' | a %}] in the BNF. Other pieces of the + OCaml AST are used as normal. *) + +(** Errors around the extension representation. These should mostly just be + fatal, but they're needed for one test case + (tests/ast-invariants/test.ml). *) +module Error : sig + (** Someone used [[%extension.EXTNAME]] wrong *) + type malformed_extension = + | Has_payload of Parsetree.payload + | Wrong_arguments of (Asttypes.arg_label * Parsetree.expression) list + | Wrong_tuple of Parsetree.pattern list + + (** An error triggered when desugaring a language extension from an OCaml AST *) + type error = + | Malformed_extension of string list * malformed_extension + | Unknown_extension of string + | Disabled_extension of Language_extension.t + | Wrong_syntactic_category of Language_extension.t * string + | Unnamed_extension + | Bad_introduction of string * string list + + (** The main exception type thrown when desugaring a language extension from an + OCaml AST; we also use the occasional [Misc.fatal_errorf]. *) + exception Error of Location.t * error +end + +(** The type of modules that lift and lower language extension terms from and + to an OCaml AST type ([ast]) *) +module type AST = sig + (** The AST type (e.g., [Parsetree.expression]) *) + type ast + + (** The name for this syntactic category in the plural form; used for error + messages (e.g., "expressions") *) + val plural : string + + (** How to get the location attached to an AST node *) + val location : ast -> Location.t + + (** Embed a language extension term in the AST with the given name + and body (the [ast]). The name will be joined with dots + and preceded by [extension.]. Partial inverse of [match_extension]. *) + val make_extension : loc:Location.t -> string list -> ast -> ast + + (** Given an AST node, check if it's a language extension term; if it is, + split it back up into its name (the [string list]) and the body (the + [ast]); the resulting name is split on dots and the leading [extension] + component is dropped. If the language extension term is malformed in any + way, raises an error; if the input isn't a language extension term, + returns [None]. Partial inverse of [make_extension]. *) + val match_extension : ast -> (string list * ast) option +end + +(** One [AST] module per syntactic category we currently care about; we're + adding these lazily as we need them. When you add another one, make + sure also to add special handling in [Ast_iterator] and [Ast_mapper]. *) + +module Expression : AST with type ast = Parsetree.expression +module Pattern : AST with type ast = Parsetree.pattern + +(** Each syntactic category will include a module that meets this signature. + Then, the [Make_of_ast] functor produces the functions that actually + convert from the Parsetree AST to the extensions one. *) +module type Of_ast_parameters = sig + + (** Which syntactic category is this concerning? e.g. [module AST = Expression] *) + module AST : AST + + (** The extension type of this syntactic category, shared across extensions. + e.g. [Extension.Expression.t] *) + type t + + (** A function to convert [Parsetree]'s AST to the extension's. + The choice of extension is extracted from the e.g. + [[%extensions.comprehensions]] node, and the argument to that + node is passed in as the [Parsetree] AST. + + So, for example, if [of_ast] spots the expression + + {[ + [%extensions.comprehensions] blah + ]} + + then it will call [of_ast_internal Comprehensions blah]. + + If the given extension does not actually extend the + syntactic category, return None; this will be reported + as an error. (Example: there are no pattern comprehensions, + so when building the pattern extension AST, this function will + return [None] when the extension in [Comprehensions].) + *) + val of_ast_internal : Language_extension.t -> AST.ast -> t option +end + +(** Build the [of_ast] function from [Of_ast_parameters]. The result + of this functor should be [include]d in modules implementing [Extensions.AST]. +*) +module Make_of_ast (Params : Of_ast_parameters) : sig + + (** Interpret an AST term in the specified syntactic category as a term of the + appropriate auxiliary language extension AST if possible. Raises an error + if the extension it finds is disabled or if the language extension + embedding is malformed. *) + val of_ast : Params.AST.ast -> Params.t option +end + +(** Require that an extension is enabled, or else throw an exception (of an + unexported type) at the provided location saying otherwise. This is + intended to be used in "extensions.ml" when a certain piece of syntax + requires two extensions to be enabled at once (e.g., immutable array + comprehensions such as [[:x for x = 1 to 10:]]). *) +val assert_extension_enabled : loc:Location.t -> Language_extension.t -> unit diff --git a/vendor/parser-extended/language_extension.ml b/vendor/parser-extended/language_extension.ml new file mode 100644 index 0000000000..d1d57629e8 --- /dev/null +++ b/vendor/parser-extended/language_extension.ml @@ -0,0 +1,200 @@ +type maturity = Stable | Beta | Alpha + +type t = + | Comprehensions + | Local + | Include_functor + | Polymorphic_parameters + | Immutable_arrays + | Module_strengthening + | Layouts of maturity + +let equal (a : t) (b : t) = (a = b) + +let all = + [ Comprehensions + ; Local + ; Include_functor + ; Polymorphic_parameters + ; Immutable_arrays + ; Module_strengthening + ; Layouts Alpha + ; Layouts Beta + ; Layouts Stable + ] + +let max_compatible = + [ Comprehensions + ; Local + ; Include_functor + ; Polymorphic_parameters + ; Immutable_arrays + ; Module_strengthening + ; Layouts Alpha + ] + +let default_extensions = + [ Local + ; Include_functor + ; Polymorphic_parameters + ] + +let to_string = function + | Comprehensions -> "comprehensions" + | Local -> "local" + | Include_functor -> "include_functor" + | Polymorphic_parameters -> "polymorphic_parameters" + | Immutable_arrays -> "immutable_arrays" + | Module_strengthening -> "module_strengthening" + | Layouts Alpha -> "layouts_alpha" + | Layouts Beta -> "layouts_beta" + | Layouts Stable -> "layouts" + +let of_string extn = match String.lowercase_ascii extn with + | "comprehensions" -> Some Comprehensions + | "local" -> Some Local + | "include_functor" -> Some Include_functor + | "polymorphic_parameters" -> Some Polymorphic_parameters + | "immutable_arrays" -> Some Immutable_arrays + | "strengthening" -> Some Module_strengthening + | "layouts_alpha" -> Some (Layouts Alpha) + | "layouts_beta" -> Some (Layouts Beta) + | "layouts" -> Some (Layouts Stable) + | _ -> None + +let of_string_exn extn = + match of_string extn with + | Some extn -> extn + | None -> raise (Arg.Bad(Printf.sprintf "Extension %s is not known" extn)) + +(* We'll do this in a more principled way later. *) +(* CR layouts: Note that layouts is only "mostly" erasable, because of annoying + interactions with the pre-layouts [@@immediate] attribute like: + + type ('a : immediate) t = 'a [@@immediate] + + But we've decided to punt on this issue in the short term. +*) +let is_erasable = function + | Local + | Layouts Alpha + | Layouts Beta + | Layouts Stable -> + true + | Comprehensions + | Include_functor + | Polymorphic_parameters + | Immutable_arrays + | Module_strengthening -> + false + +module Universe = struct + (** Which extensions can be enabled? *) + type t = + | No_extensions + | Only_erasable + | Any + + let compiler_options = function + | No_extensions -> "flag -disable-all-extensions" + | Only_erasable -> "flag -only-erasable-extensions" + | Any -> "default options" + + let is_allowed t ext = match t with + | No_extensions -> false + | Only_erasable -> is_erasable ext + | Any -> true +end + +(* Mutable state. Invariants: + + (1) [!extensions] contains at most one copy of each extension. + + (2) Every member of [!extensions] satisfies [Universe.is_allowed !universe]. + (For instance, [!universe = No_extensions] implies + [!extensions = []]). *) +let extensions = ref default_extensions (* -extension *) +let universe = ref Universe.Any (* -only-erasable-extensions, + -disable-all-extensions *) + +type compatibility = Compatible | Duplicate | Incompatible of string + +let check_conflicts t1 = + let layouts_err = + "Invalid extensions: Please enable at most one of 'layouts', \ + 'layouts_beta', and 'layouts_alpha'." + in + let c = List.find_map (fun t2 -> + if t1 = t2 then Some Duplicate else + match t1, t2 with + | Layouts _, Layouts _ -> Some (Incompatible layouts_err) + | _, _ -> None) + !extensions + in + Option.value c ~default:Compatible + +let set extn ~enabled = + if enabled then begin + if not (Universe.is_allowed !universe extn) then + raise (Arg.Bad(Printf.sprintf + "Cannot %s extension %s: incompatible with %s" + (if enabled then "enable" else "disable") + (to_string extn) + (Universe.compiler_options !universe))); + match check_conflicts extn with + | Duplicate -> () + | Compatible -> extensions := extn :: !extensions + | Incompatible err -> raise (Arg.Bad err) + end else + extensions := + List.filter (fun extn' -> + match extn, extn' with + | Layouts _, Layouts _ -> + raise (Arg.Bad(Printf.sprintf + "Cannot disable extension %s because extension %s is enabled. \ + Please enable or disable at most one of the layouts extensions." + (to_string extn) (to_string extn'))) + | _, _ -> not (equal extn extn')) + !extensions + +let enable = set ~enabled:true +let disable = set ~enabled:false + +let is_enabled extn = List.mem extn !extensions + +(* It might make sense to ban [set], [enable], [disable], + [only_erasable_extensions], and [disallow_extensions] inside [f], but it's + not clear that it's worth the hassle *) +let with_set extn ~enabled f = + (* This is similar to [Misc.protect_refs], but we don't have values to set + [extensions] and [universe] to. *) + let current_extensions = !extensions in + let current_universe = !universe in + Fun.protect + ~finally:(fun () -> + extensions := current_extensions; + universe := current_universe) + (fun () -> + set extn ~enabled; + f ()) + +let with_enabled = with_set ~enabled:true +let with_disabled = with_set ~enabled:false + +let restrict_to_erasable_extensions () = + match !universe with + | Any -> + extensions := List.filter is_erasable !extensions; + universe := Universe.Only_erasable + | Only_erasable -> + () (* Idempotent *) + | No_extensions -> + raise (Arg.Bad(Printf.sprintf + "Cannot specify %s: incompatible with %s" + (Universe.compiler_options Only_erasable) + (Universe.compiler_options No_extensions))) + +let disallow_extensions () = + (* The strictest option, so no internal checks needed *) + extensions := []; + universe := Universe.No_extensions diff --git a/vendor/parser-extended/language_extension.mli b/vendor/parser-extended/language_extension.mli new file mode 100644 index 0000000000..f48e47985e --- /dev/null +++ b/vendor/parser-extended/language_extension.mli @@ -0,0 +1,73 @@ +(** Language extensions provided by ocaml-jst *) + +type maturity = Stable | Beta | Alpha + +(** The type of language extensions *) +type t = + | Comprehensions + | Local + | Include_functor + | Polymorphic_parameters + | Immutable_arrays + | Module_strengthening + | Layouts of maturity + +(** Equality on language extensions *) +val equal : t -> t -> bool + +(** A list of all possible language extensions *) +val all : t list + +(** A maximal list of compatible language extensions (of the layouts extensions, + "layouts_alpha" is selected). *) +val max_compatible : t list + +(** Check if a language extension is "erasable", i.e. whether it can be + harmlessly translated to attributes and compiled with the upstream + compiler. *) +val is_erasable : t -> bool + +(** Print and parse language extensions; parsing is case-insensitive *) +val to_string : t -> string +val of_string : string -> t option +val of_string_exn : string -> t + +(** Enable and disable language extensions; these operations are idempotent *) +val set : t -> enabled:bool -> unit +val enable : t -> unit +val disable : t -> unit + +(** Check if a language extension is currently enabled *) +val is_enabled : t -> bool + +(** Tooling support: Temporarily enable and disable language extensions; these + operations are idempotent. Calls to [set], [enable], [disable], and + [disallow_extensions] inside the body of the function argument will also + be rolled back when the function finishes, but this behavior may change; + nest multiple [with_*] functions instead. *) +val with_set : t -> enabled:bool -> (unit -> unit) -> unit +val with_enabled : t -> (unit -> unit) -> unit +val with_disabled : t -> (unit -> unit) -> unit + +(** Permanently restrict the allowable extensions to those that are + "erasable", i.e. those that can be harmlessly translated to attributes and + compiled with the upstream compiler. Used for [-only-erasable-extensions] + to ensure that some code is guaranteed to be compatible with upstream + OCaml after rewriting to attributes. When called, disables any + currently-enabled non-erasable extensions, including any that are on by + default. Causes any future uses of [set ~enabled:true], [enable], and + their [with_] variants to raise if used with a non-erasable extension. + The [is_enabled] function will still work on any extensions, it will just + always return [false] on non-erasable ones. Will raise if called after + [disallow_extensions]; the ratchet of extension restriction only goes one + way. *) +val restrict_to_erasable_extensions : unit -> unit + +(** Permanently ban all extensions; used for [-disable-all-extensions] to + ensure that some code is 100% extension-free. When called, disables any + currently-enabled extensions, including the defaults. Causes any future + uses of [set ~enabled:true], [enable], and their [with_] variants to + raise; also causes any future uses of [only_erasable_extensions] to raise. + The [is_enabled] function will still work, it will just always return + [false].*) +val disallow_extensions : unit -> unit diff --git a/vendor/parser-extended/lexer.mll b/vendor/parser-extended/lexer.mll index c6713eca47..a71c4f118c 100644 --- a/vendor/parser-extended/lexer.mll +++ b/vendor/parser-extended/lexer.mll @@ -50,12 +50,14 @@ let keyword_table = "else", ELSE; "end", END; "exception", EXCEPTION; + "exclave_", EXCLAVE; "external", EXTERNAL; "false", FALSE; "for", FOR; "fun", FUN; "function", FUNCTION; "functor", FUNCTOR; + "global_", GLOBAL; "if", IF; "in", IN; "include", INCLUDE; @@ -63,6 +65,7 @@ let keyword_table = "initializer", INITIALIZER; "lazy", LAZY; "let", LET; + "local_", LOCAL; "match", MATCH; "method", METHOD; "module", MODULE; @@ -117,6 +120,118 @@ let is_in_string = ref false let in_string () = !is_in_string let print_warnings = ref true +(* Jane Street extension *) +let at_beginning_of_line pos = (pos.pos_cnum = pos.pos_bol) + +(* See the comment on the [directive] lexer. *) +type directive_lexing_already_consumed = + | Hash + | Hash_and_line_num of { line_num : string } + +type deferred_token = + { token : token + ; start_pos : Lexing.position + ; end_pos : Lexing.position + } + +(* This queue will only ever have 0 or 1 elements in it. We use it + instead of an [option ref] for its convenient interface. +*) +let deferred_tokens : deferred_token Queue.t = Queue.create () + +(* Effectively splits the text in the lexer's current "window" (defined below) + into two halves. The current call to the lexer will return the first half of + the text in the window, and the next call to the lexer will return the second + half (of length [len]) of the text in the window. + + "window" refers to the text matched by a production of the lexer. It spans + from [lexer.lex_start_p] to [lexer.lex_curr_p]. + + The function accomplishes this splitting by doing two things: + - It sets the current window of the lexbuf to only account for the + first half of the text. (The first half is of length: |text|-len.) + - It enqueues a token into [deferred_tokens] such that, the next time the + lexer is called, it will return the specified [token] *and* set the window + of the lexbuf to account for the second half of the text. (The second half + is of length: |text|.) + + This business with setting the window of the lexbuf is only so that error + messages point at the right place in the program text. +*) +let enqueue_token_from_end_of_lexbuf_window (lexbuf : Lexing.lexbuf) token ~len = + let suffix_end = lexbuf.lex_curr_p in + let suffix_start = + { suffix_end with pos_cnum = suffix_end.pos_cnum - len } + in + lexbuf.lex_curr_p <- suffix_start; + Queue.add + { token; start_pos = suffix_start; end_pos = suffix_end } + deferred_tokens + +(* Note [Lexing hack for float#] + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + This note describes a non-backward-compatible Jane Street--internal change to + the lexer. + + We want the lexer to lex [float#] differently than [float #]. [float#] is the + new syntax for the unboxed float type. It veers close to the syntax for the + type of all objects belonging to a class [c], which is [#c]. The way we + navigate this veering is by producing the following tokens for these source + program examples, where LIDENT(s) is an LIDENT with text [s]. + + float#c ==> LIDENT(float) HASH_SUFFIX LIDENT(c) + float# c ==> LIDENT(float) HASH_SUFFIX LIDENT(c) + float # c ==> LIDENT(float) HASH LIDENT(c) + float #c ==> LIDENT(float) HASH LIDENT(c) + + (A) The parser interprets [LIDENT(float) HASH_SUFFIX LIDENT(c)] as + "the type constructor [c] applied to the unboxed float type." + (B) The parser interprets [LIDENT(float) HASH LIDENT(c)] as + "the type constructor [#c] applied to the usual boxed float type." + + This is not a backward-compatible change. In upstream ocaml, the lexer + produces [LIDENT(float) HASH LIDENT(c)] for all the above source programs. + + But, this isn't problematic: everybody puts a space before '#c' to mean (B). + No existing code writes things like [float#c] or indeed [float# c]. + + We accomplish this hack by setting some global mutable state upon seeing + an identifier immediately followed by a hash. When that state is set, we + will produce [HASH_SUFFIX] the next time the lexer is called. This is + done in [enqueue_hash_suffix_from_end_of_lexbuf_window]. + + Note [Lexing hack for hash operators] + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + To complicate the above story, we don't want to treat the # in the + below program as HASH_SUFFIX: + + x#~#y + + We instead want: + + x#~#y ==> LIDENT(x) HASHOP(#~#) LIDENT(y) + + This is to allow for infix hash operators. We add an additional hack, in + the style of Note [Lexing hack for float#], where the lexer consumes [x#~#] + all at once, but produces LIDENT(x) from the current call to the lexer and + HASHOP(#~#) from the next call to the lexer. This is done in + [enqueue_hashop_from_end_of_lexbuf_window]. + *) + +let enqueue_hash_suffix_from_end_of_lexbuf_window lexbuf = + enqueue_token_from_end_of_lexbuf_window lexbuf HASH_SUFFIX ~len:1 + +let enqueue_hashop_from_end_of_lexbuf_window lexbuf ~hashop = + enqueue_token_from_end_of_lexbuf_window lexbuf (HASHOP hashop) + ~len:(String.length hashop) + +let lookup_keyword name = + match Hashtbl.find keyword_table name with + | kw -> kw + | exception Not_found -> + LIDENT name +(* End Jane Street extension *) + (* Escaped chars are interpreted in strings unless they are in comments. *) let store_escaped_char lexbuf c = if in_comment () then store_lexeme lexbuf else store_string_char c @@ -159,6 +274,27 @@ let wrap_comment_lexer comment lexbuf = let error lexbuf e = raise (Error(e, Location.curr lexbuf)) let error_loc loc e = raise (Error(e, loc)) +(* Jane Street extension *) +let directive_error + (lexbuf : Lexing.lexbuf) explanation ~directive ~already_consumed + = + let directive_prefix = + match already_consumed with + | Hash -> "#" + | Hash_and_line_num { line_num } -> "#" ^ line_num + in + (* Set the lexbuf's current window to extend to the start of + the directive so the error message's location is more accurate. + *) + lexbuf.lex_start_p <- + { lexbuf.lex_start_p with + pos_cnum = + lexbuf.lex_start_p.pos_cnum - String.length directive_prefix + }; + error lexbuf + (Invalid_directive (directive_prefix ^ directive, Some explanation)) +(* End Jane Street extension *) + (* to translate escape sequences *) let digit_value c = @@ -304,6 +440,20 @@ let add_docstring_comment ds = let comments () = List.rev !comment_list +(* Jane Street extension *) +let float ~maybe_hash lit modifier = + match maybe_hash with + | "#" -> HASH_FLOAT (lit, modifier) + | "" -> FLOAT (lit, modifier) + | unexpected -> fatal_error ("expected # or empty string: " ^ unexpected) + +let int ~maybe_hash lit modifier = + match maybe_hash with + | "#" -> HASH_INT (lit, modifier) + | "" -> INT (lit, modifier) + | unexpected -> fatal_error ("expected # or empty string: " ^ unexpected) +(* End Jane Street extension *) + (* Error report *) open Format @@ -436,24 +586,75 @@ rule token = parse | "?" (lowercase_latin1 identchar_latin1 * as name) ':' { warn_latin1 lexbuf; OPTLABEL name } + + (* Jane Street extension *) + (* Lowercase identifiers are split into 3 cases, and the order matters + (longest to shortest). + *) + | (lowercase identchar * as name) ('#' symbolchar_or_hash+ as hashop) + (* See Note [Lexing hack for hash operators] *) + { enqueue_hashop_from_end_of_lexbuf_window lexbuf ~hashop; + lookup_keyword name } + | (lowercase identchar * as name) '#' + (* See Note [Lexing hack for float#] *) + { enqueue_hash_suffix_from_end_of_lexbuf_window lexbuf; + lookup_keyword name } + (* End Jane Street extension *) + | lowercase identchar * as name { try Hashtbl.find keyword_table name with Not_found -> LIDENT name } + + (* Jane Street extension *) + (* Lowercase latin1 identifiers are split into 3 cases, and the order matters + (longest to shortest). + *) + | (lowercase_latin1 identchar_latin1 * as name) + ('#' symbolchar_or_hash+ as hashop) + (* See Note [Lexing hack for hash operators] *) + { warn_latin1 lexbuf; + enqueue_hashop_from_end_of_lexbuf_window lexbuf ~hashop; + LIDENT name } + | (lowercase_latin1 identchar_latin1 * as name) '#' + (* See Note [Lexing hack for float#] *) + { warn_latin1 lexbuf; + enqueue_hash_suffix_from_end_of_lexbuf_window lexbuf; + LIDENT name } + (* End Jane Street extension *) + | lowercase_latin1 identchar_latin1 * as name { warn_latin1 lexbuf; LIDENT name } | uppercase identchar * as name { UIDENT name } (* No capitalized keywords *) | uppercase_latin1 identchar_latin1 * as name { warn_latin1 lexbuf; UIDENT name } - | int_literal as lit { INT (lit, None) } - | (int_literal as lit) (literal_modifier as modif) - { INT (lit, Some modif) } - | float_literal | hex_float_literal as lit - { FLOAT (lit, None) } - | (float_literal | hex_float_literal as lit) (literal_modifier as modif) - { FLOAT (lit, Some modif) } - | (float_literal | hex_float_literal | int_literal) identchar+ as invalid + + (* Jane Street modification *) + (* This matches either an integer literal or a directive. If the text "#2" + appears at the beginning of a line that lexes as a directive, then it + should be treated as a directive and not an unboxed int. This is acceptable + because "#2" isn't a valid unboxed int anyway because it lacks a suffix; + the parser rejects unboxed-ints-lacking-suffixes with a more descriptive + error message. + *) + | ('#'? as maybe_hash) (int_literal as lit) + { if at_beginning_of_line lexbuf.lex_start_p && maybe_hash = "#" then + try directive (Hash_and_line_num { line_num = lit }) lexbuf + with Failure _ -> int ~maybe_hash lit None + else int ~maybe_hash lit None + } + | ('#'? as maybe_hash) (int_literal as lit) (literal_modifier as modif) + { int ~maybe_hash lit (Some modif) } + | ('#'? as maybe_hash) + (float_literal | hex_float_literal as lit) + { float ~maybe_hash lit None } + | ('#'? as maybe_hash) + (float_literal | hex_float_literal as lit) (literal_modifier as modif) + { float ~maybe_hash lit (Some modif) } + | '#'? (float_literal | hex_float_literal | int_literal) identchar+ as invalid { error lexbuf (Invalid_literal invalid) } + (* End Jane Street modification *) + | "\"" { let s, loc = wrap_string_lexer string lexbuf in STRING (s, loc, None) } @@ -536,12 +737,15 @@ rule token = parse lexbuf.lex_curr_p <- { curpos with pos_cnum = curpos.pos_cnum - 1 }; STAR } + + (* Jane Street modification *) | "#" - { let at_beginning_of_line pos = (pos.pos_cnum = pos.pos_bol) in - if not (at_beginning_of_line lexbuf.lex_start_p) + { if not (at_beginning_of_line lexbuf.lex_start_p) then HASH - else try directive lexbuf with Failure _ -> HASH + else try directive Hash lexbuf with Failure _ -> HASH } + (* End Jane Street modification *) + | "&" { AMPERSAND } | "&&" { AMPERAMPER } | "`" { BACKQUOTE } @@ -565,6 +769,7 @@ rule token = parse | "=" { EQUAL } | "[" { LBRACKET } | "[|" { LBRACKETBAR } + | "[:" { LBRACKETCOLON } | "[<" { LBRACKETLESS } | "[>" { LBRACKETGREATER } | "]" { RBRACKET } @@ -573,6 +778,7 @@ rule token = parse | "|" { BAR } | "||" { BARBAR } | "|]" { BARRBRACKET } + | ":]" { COLONRBRACKET } | ">" { GREATER } | ">]" { GREATERRBRACKET } | "}" { RBRACE } @@ -616,15 +822,35 @@ rule token = parse | (_ as illegal_char) { error lexbuf (Illegal_character illegal_char) } -and directive = parse - | ([' ' '\t']* (['0'-'9']+ as _num) [' ' '\t']* - ("\"" ([^ '\010' '\013' '\"' ] * as _name) "\"") as directive) +(* Jane Street modification *) +(* An example of a directive is: + +#4 "filename.ml" + + Here, 4 is the line number and filename.ml is the file name. The '#' must + appear in column 0. + + The [directive] lexer is called when some portion of the start of + the line was already consumed, either just the '#' or the '#4'. That's + indicated by the [already_consumed] argument. The caller is responsible + for checking that the '#' appears in column 0. + + The [directive] lexer always attempts to read the line number from the + lexbuf. It expects to receive a line number from exactly one source (either + the lexbuf or the [already_consumed] argument, but not both) and will fail if + this isn't the case. +*) +and directive already_consumed = parse + | ([' ' '\t']* (['0'-'9']+? as _line_num_opt) [' ' '\t']* + ("\"" ([^ '\010' '\013' '\"' ] * as _name) "\"") as directive) [^ '\010' '\013'] * { (* Line directives are not preserved by the lexer so we error out. *) let explanation = "line directives are not supported" in - error lexbuf (Invalid_directive ("#" ^ directive, Some explanation)) + directive_error lexbuf explanation ~already_consumed ~directive } +(* End Jane Street modification *) + and comment = parse "(*" { comment_start_loc := (Location.curr lexbuf) :: !comment_start_loc; @@ -784,6 +1010,15 @@ and skip_hash_bang = parse | "" { () } { + (* Jane Street extension *) + let token lexbuf = + match Queue.take_opt deferred_tokens with + | None -> token lexbuf + | Some { token; start_pos; end_pos } -> + lexbuf.lex_start_p <- start_pos; + lexbuf.lex_curr_p <- end_pos; + token + (* End Jane Street extension *) let token_with_comments lexbuf = match !preprocessor with diff --git a/vendor/parser-extended/parser.mly b/vendor/parser-extended/parser.mly index 26673cdfae..b30cbabbf8 100644 --- a/vendor/parser-extended/parser.mly +++ b/vendor/parser-extended/parser.mly @@ -160,6 +160,12 @@ let ghsig ~loc d = Sig.mk ~loc:(ghost_loc loc) d let mkinfix arg1 op arg2 = Pexp_infix(op, arg1, arg2) +(* Jane Street extension *) +let flip_sign = function + | Positive -> Negative + | Negative -> Positive +(* End Jane Street extension *) + let neg_string f = if String.length f > 0 && f.[0] = '-' then String.sub f 1 (String.length f - 1) @@ -171,6 +177,17 @@ let mkuminus ~oploc name arg = Pexp_constant({c with pconst_desc= Pconst_integer(neg_string n,m)}) | ("-" | "-."), Pexp_constant({pconst_desc= Pconst_float (f, m); _} as c) -> Pexp_constant({c with pconst_desc= Pconst_float(neg_string f, m)}) + + (* Jane Street extension *) + | "-", Pexp_constant({pconst_desc= Pconst_unboxed_integer (s,n,m); _} as c) -> + Pexp_constant({c with + pconst_desc= Pconst_unboxed_integer(flip_sign s,n,m)}) + | ("-" | "-."), Pexp_constant({pconst_desc= + Pconst_unboxed_float (s, f, m); _} as c) -> + Pexp_constant({c with + pconst_desc= Pconst_unboxed_float(flip_sign s, f, m)}) + (* End Jane Street extension *) + | _ -> Pexp_prefix(mkoperator ~loc:oploc ("~" ^ name), arg) @@ -179,9 +196,105 @@ let mkuplus ~oploc name arg = match name, desc with | "+", Pexp_constant({pconst_desc= Pconst_integer _; _}) | ("+" | "+."), Pexp_constant({pconst_desc= Pconst_float _; _}) -> desc + + (* Jane Street extension *) + | "+", Pexp_constant({pconst_desc= Pconst_unboxed_integer _; _}) + | ("+" | "+."), Pexp_constant({pconst_desc= Pconst_unboxed_float _; _}) + -> desc + (* End Jane Street extension *) + | _ -> Pexp_prefix(mkoperator ~loc:oploc ("~" ^ name), arg) + +let local_ext_loc = mknoloc "extension.local" + +let local_attr = + Attr.mk ~loc:Location.none local_ext_loc (PStr []) + +let local_extension = + Exp.mk ~loc:Location.none (Pexp_extension(local_ext_loc, PStr [])) + +let include_functor_ext_loc = mknoloc "extension.include_functor" + +let include_functor_attr = + Attr.mk ~loc:Location.none include_functor_ext_loc (PStr []) + +let mkexp_stack ~loc exp = + if Erase_jane_syntax.should_erase () then exp else + ghexp ~loc (Pexp_apply(local_extension, [Nolabel, exp])) + +let mkpat_stack pat = + if Erase_jane_syntax.should_erase () then pat else + {pat with ppat_attributes = local_attr :: pat.ppat_attributes} + +let mktyp_stack typ = + if Erase_jane_syntax.should_erase () then typ else + {typ with ptyp_attributes = local_attr :: typ.ptyp_attributes} + +let wrap_exp_stack exp = + if Erase_jane_syntax.should_erase () then exp else + {exp with pexp_attributes = local_attr :: exp.pexp_attributes} + +let mkexp_local_if p ~loc exp = + if p then mkexp_stack ~loc exp else exp + +let mkpat_local_if p pat = + if p then mkpat_stack pat else pat + +let mktyp_local_if p typ = + if p then mktyp_stack typ else typ + +let exclave_ext_loc loc = mkloc "extension.exclave" loc + +let exclave_extension loc = + Exp.mk ~loc:Location.none + (Pexp_extension(exclave_ext_loc loc, PStr [])) + +let mkexp_exclave ~loc ~kwd_loc exp = + if Erase_jane_syntax.should_erase () then exp else + ghexp ~loc (Pexp_apply(exclave_extension (make_loc kwd_loc), [Nolabel, exp])) + +let curry_attr = + Attr.mk ~loc:Location.none (mknoloc "extension.curry") (PStr []) + +let is_curry_attr attr = + attr.attr_name.txt = "extension.curry" + +let mktyp_curry typ = + if Erase_jane_syntax.should_erase () then typ else + {typ with ptyp_attributes = curry_attr :: typ.ptyp_attributes} + +let maybe_curry_typ typ = + match typ.ptyp_desc with + | Ptyp_arrow _ -> + if List.exists is_curry_attr typ.ptyp_attributes then typ + else mktyp_curry typ + | _ -> typ + +let global_loc loc = mkloc "extension.global" loc + +let global_attr loc = + Attr.mk ~loc:Location.none (global_loc loc) (PStr []) + +let mkld_global ld loc = + if Erase_jane_syntax.should_erase () then ld else + { ld with pld_attributes = global_attr loc :: ld.pld_attributes } + +let mkld_global_maybe gbl ld loc = + match gbl with + | Global -> mkld_global ld loc + | Nothing -> ld + +let mkcty_global cty loc = + if Erase_jane_syntax.should_erase () then cty else + { cty with ptyp_attributes = global_attr loc :: cty.ptyp_attributes } + +let mkcty_global_maybe gbl cty loc = + match gbl with + | Global -> mkcty_global cty loc + | Nothing -> cty + (* TODO define an abstraction boundary between locations-as-pairs and locations-as-Location.t; it should be clear when we move from one world to the other *) @@ -249,6 +362,50 @@ let unclosed opening_name opening_loc closing_name closing_loc = raise(Syntaxerr.Error(Syntaxerr.Unclosed(make_loc opening_loc, opening_name, make_loc closing_loc, closing_name))) +(* Normal mutable arrays and immutable arrays are parsed identically, just with + different delimiters. The parsing is done by the [array_exprs] rule, and the + [Generic_array] module provides (1) a type representing the possible results, + and (2) a function for going from that type to an AST fragment representing + an array. *) +module Generic_array = struct + (** The three possible ways to parse an array (writing [[? ... ?]] for either + [[| ... |]] or [[: ... :]]): *) + type (_, _) t = + | Literal : 'ast list -> ('ast, 'ast_desc) t + (** A plain array literal/pattern, [[? x; y; z ?]] *) + | Opened_literal : Longident.t Location.loc * + Lexing.position * + Lexing.position * + expression list + -> (expression, expression_desc) t + (** An array literal with a local open, [Module.[? x; y; z ?]] (only valid in + expressions) *) + | Unclosed : (Lexing.position * Lexing.position) * + (Lexing.position * Lexing.position) + -> (_, _) t + (** Parse error: an unclosed array literal, [\[? x; y; z] with no closing + [?\]]. *) + + let to_ast (type ast ast_desc) + (open_ : string) (close : string) + (array : ast list -> ast_desc) + : (ast, ast_desc) t -> ast_desc = function + | Literal elts -> + array elts + | Opened_literal(od, startpos, endpos, elts) -> + (Pexp_open(od, mkexp ~loc:(startpos, endpos) (array elts)) : ast_desc) + | Unclosed(startpos, endpos) -> + unclosed open_ startpos close endpos + + let expression : _ -> _ -> _ -> (expression, expression_desc) t -> _ = to_ast + let pattern : _ -> _ -> _ -> (pattern, pattern_desc) t -> _ = to_ast +end + +let ppat_iarray loc elts = + (Extensions.Immutable_arrays.pat_of + ~loc:(make_loc loc) + (Iapat_immutable_array elts)).ppat_desc + let expecting loc nonterm = raise Syntaxerr.(Error(Expecting(make_loc loc, nonterm))) @@ -539,6 +696,14 @@ let mk_directive ~loc name arg = pdir_loc = make_loc loc; } +let check_layout loc id = + begin + match id with + | ("any" | "value" | "void" | "immediate64" | "immediate" | "float64") -> () + | _ -> expecting loc "layout" + end; + let loc = make_loc loc in + Attr.mk ~loc (mkloc id loc) (PStr []) %} /* Tokens */ @@ -569,6 +734,7 @@ let mk_directive ~loc name arg = %token COLONCOLON "::" %token COLONEQUAL ":=" %token COLONGREATER ":>" +%token COLONRBRACKET ":]" %token COMMA "," %token CONSTRAINT "constraint" %token DO "do" @@ -581,6 +747,7 @@ let mk_directive ~loc name arg = %token EOF "" %token EQUAL "=" %token EXCEPTION "exception" +%token EXCLAVE "exclave_" %token EXTERNAL "external" %token FALSE "false" %token FLOAT "42.0" (* just an example *) @@ -588,6 +755,7 @@ let mk_directive ~loc name arg = %token FUN "fun" %token FUNCTION "function" %token FUNCTOR "functor" +%token GLOBAL "global_" %token GREATER ">" %token GREATERRBRACE ">}" %token GREATERRBRACKET ">]" @@ -611,6 +779,7 @@ let mk_directive ~loc name arg = %token LBRACELESS "{<" %token LBRACKET "[" %token LBRACKETBAR "[|" +%token LBRACKETCOLON "[:" %token LBRACKETLESS "[<" %token LBRACKETGREATER "[>" %token LBRACKETPERCENT "[%" @@ -619,6 +788,7 @@ let mk_directive ~loc name arg = %token LESSMINUS "<-" %token LET "let" %token LIDENT "lident" (* just an example *) +%token LOCAL "local_" %token LPAREN "(" %token LBRACKETAT "[@" %token LBRACKETATAT "[@@" @@ -684,6 +854,12 @@ let mk_directive ~loc name arg = %token TYPE_DISAMBIGUATOR "2" (* just an example *) +(* Jane Street extension *) +%token HASH_FLOAT "#42.0" (* just an example *) +%token HASH_INT "#42l" (* just an example *) +%token HASH_SUFFIX "# " +(* End Jane Street extension *) + /* Precedences and associativities. Tokens and rules have precedences. A reduce/reduce conflict is resolved @@ -710,7 +886,7 @@ The precedences must be listed from low to high. %nonassoc IN %nonassoc below_SEMI %nonassoc SEMI /* below EQUAL ({lbl=...; lbl=...}) */ -%nonassoc LET /* above SEMI ( ...; let ... in ...) */ +%nonassoc LET FOR /* above SEMI ( ...; let ... in ...) */ %nonassoc below_WITH %nonassoc FUNCTION WITH /* below BAR (match ... with ...) */ %nonassoc AND /* above WITH (module rec A: SIG with ... and ...) */ @@ -722,6 +898,8 @@ The precedences must be listed from low to high. %left BAR /* pattern (p|p|p) */ %nonassoc below_COMMA %left COMMA /* expr/expr_comma_list (e,e,e) */ +%nonassoc below_FUNCTOR /* include M */ +%nonassoc FUNCTOR /* include functor M */ %right MINUSGREATER /* function_type (t -> t -> t) */ %right OR BARBAR /* expr (e || e || e) */ %right AMPERSAND AMPERAMPER /* expr (e && e && e) */ @@ -738,13 +916,13 @@ The precedences must be listed from low to high. %nonassoc prec_constant_constructor /* cf. simple_expr (C versus C x) */ %nonassoc prec_constr_appl /* above AS BAR COLONCOLON COMMA */ %nonassoc below_HASH -%nonassoc HASH /* simple_expr/toplevel_directive */ +%nonassoc HASH HASH_SUFFIX /* simple_expr/toplevel_directive */ %left HASHOP %nonassoc below_DOT %nonassoc DOT DOTOP /* Finally, the first tokens of simple_expr are above everything else. */ -%nonassoc BACKQUOTE BANG BEGIN CHAR FALSE FLOAT INT OBJECT - LBRACE LBRACELESS LBRACKET LBRACKETBAR LIDENT LPAREN +%nonassoc BACKQUOTE BANG BEGIN CHAR FALSE FLOAT HASH_FLOAT INT HASH_INT OBJECT + LBRACE LBRACELESS LBRACKET LBRACKETBAR LBRACKETCOLON LIDENT LPAREN NEW PREFIXOP STRING TRUE UIDENT UNDERSCORE LBRACKETPERCENT QUOTED_STRING_EXPR @@ -1450,16 +1628,23 @@ module_binding_body: (* Shared material between structures and signatures. *) +include_and_functor_attr: + | INCLUDE %prec below_FUNCTOR + { [] } + | INCLUDE FUNCTOR + { [include_functor_attr] } +; + (* An [include] statement can appear in a structure or in a signature, which is why this definition is parameterized. *) %inline include_statement(thing): - INCLUDE + attrs0 = include_and_functor_attr ext = ext attrs1 = attributes thing = thing attrs2 = post_item_attributes { - let attrs = attrs1 @ attrs2 in + let attrs = attrs0 @ attrs1 @ attrs2 in let loc = make_loc $sloc in let docs = symbol_docs $sloc in Incl.mk thing ~attrs ~loc ~docs, ext @@ -1565,6 +1750,8 @@ module_type: { Pmty_with($1, $3) } /* | LPAREN MODULE mkrhs(mod_longident) RPAREN { Pmty_alias $3 } */ + | module_type WITH mkrhs(mod_ext_longident) + { Pmty_strengthen($1,$3) } | extension { Pmty_extension $1 } ) @@ -2149,22 +2336,32 @@ seq_expr: mkexp ~loc:$sloc (Pexp_extension ($4, payload)) } ; labeled_simple_pattern: - QUESTION LPAREN label_let_pattern opt_default RPAREN - { mk_optional (fst $3) $sloc, $4, snd $3 } + QUESTION LPAREN optional_local label_let_pattern opt_default RPAREN + { mk_optional (fst $4) $sloc, $5, mkpat_local_if $3 (snd $4) } | QUESTION label_var { mk_optional (fst $2) $sloc, None, snd $2 } - | OPTLABEL LPAREN let_pattern opt_default RPAREN - { mk_optional $1 $sloc, $4, $3 } + | OPTLABEL LPAREN optional_local let_pattern opt_default RPAREN + { mk_optional $1 $sloc, $5, mkpat_local_if $3 $4 } | OPTLABEL pattern_var { mk_optional $1 $sloc, None, $2 } - | TILDE LPAREN label_let_pattern RPAREN - { mk_labelled (fst $3) $sloc, None, snd $3 } + | TILDE LPAREN optional_local label_let_pattern RPAREN + { mk_labelled (fst $4) $sloc, None, mkpat_local_if $3 (snd $4) } | TILDE label_var { mk_labelled (fst $2) $sloc, None, snd $2 } | LABEL simple_pattern { mk_labelled $1 $sloc, None, $2 } + | LABEL LPAREN LOCAL pattern RPAREN + { mk_labelled $1 $sloc, None, mkpat_stack $4 } | simple_pattern { Nolabel, None, $1 } + | LPAREN LOCAL let_pattern RPAREN + { Nolabel, None, mkpat_stack $3 } + | LABEL LPAREN poly_pattern RPAREN + { mk_labelled $1 $sloc, None, $3 } + | LABEL LPAREN LOCAL poly_pattern RPAREN + { mk_labelled $1 $sloc, None, mkpat_stack $4 } + | LPAREN poly_pattern RPAREN + { Nolabel, None, $2 } ; pattern_var: @@ -2185,6 +2382,11 @@ label_let_pattern: { let lab, pat = x in lab, mkpat ~loc:$sloc (Ppat_constraint (pat, cty)) } + | x = label_var COLON + cty = mktyp (vars = typevar_list DOT ty = core_type { Ptyp_poly(vars, ty) }) + { let lab, pat = x in + lab, + mkpat ~loc:$sloc (Ppat_constraint (pat, cty)) } ; %inline label_var: mkrhs(LIDENT) @@ -2196,6 +2398,17 @@ let_pattern: | mkpat(pattern COLON core_type { Ppat_constraint($1, $3) }) { $1 } + | poly_pattern + { $1 } +; +%inline poly_pattern: + mkpat( + pat = pattern + COLON + cty = mktyp(vars = typevar_list DOT ty = core_type + { Ptyp_poly(vars, ty) }) + { Ppat_constraint(pat, cty) }) + { $1 } ; %inline indexop_expr(dot, index, right): @@ -2254,6 +2467,10 @@ expr: { not_expecting $loc($1) "wildcard \"_\"" } *) /* END AVOID */ + | LOCAL seq_expr + { mkexp_stack ~loc:$sloc $2 } + | EXCLAVE seq_expr + { mkexp_exclave ~loc:$sloc ~kwd_loc:($loc($1)) $2 } ; %inline expr_attrs: | LET MODULE ext_attributes mkrhs(module_name) functor_args module_binding_body IN seq_expr @@ -2267,10 +2484,12 @@ expr: | FUNCTION ext_attributes match_cases { Pexp_function $3, $2 } | FUN ext_attributes labeled_simple_pattern fun_def - { let (l,o,p) = $3 in - Pexp_fun(l, o, p, $4), $2 } + { let ext, attrs = $2 in + let (l,o,p) = $3 in + Pexp_fun(l, o, p, $4), (ext, attrs) } | FUN ext_attributes LPAREN TYPE lident_list RPAREN fun_def - { (mk_newtypes ~loc:$sloc $5 $7).pexp_desc, $2 } + { let ext, attrs = $2 in + (mk_newtypes ~loc:$sloc $5 $7).pexp_desc, (ext, attrs) } | MATCH ext_attributes seq_expr WITH match_cases { Pexp_match($3, $5), $2 } | TRY ext_attributes seq_expr WITH match_cases @@ -2335,6 +2554,8 @@ simple_expr: { mkexp_constraint ~loc:$sloc $2 $3 } | indexop_expr(DOT, seq_expr, { None }) { mk_builtin_indexop_expr ~loc:$sloc $1 } + (* Immutable array indexing is a regular operator, so it doesn't need its own + rule and is handled by the next case *) | indexop_expr(qualified_dotop, expr_semi_list, { None }) { mk_dotop_indexop_expr ~loc:$sloc $1 } | indexop_error (DOT, seq_expr) { $1 } @@ -2365,6 +2586,91 @@ simple_expr: | OBJECT ext_attributes class_structure error { unclosed "object" $loc($1) "end" $loc($4) } ; + +comprehension_iterator: + | EQUAL expr direction_flag expr + { Extensions.Comprehensions.Range { start = $2 ; stop = $4 ; direction = $3 } } + | IN expr + { Extensions.Comprehensions.In $2 } +; + +comprehension_clause_binding: + | attributes pattern comprehension_iterator + { Extensions.Comprehensions.{ pattern = $2 ; iterator = $3 ; attributes = $1 } } + (* We can't write [[e for local_ x = 1 to 10]], because the [local_] has to + move to the RHS and there's nowhere for it to move to; besides, you never + want that [int] to be [local_]. But we can parse [[e for local_ x in xs]]. + We have to have that as a separate rule here because it moves the [local_] + over to the RHS of the binding, so we need everything to be visible. *) + | attributes LOCAL pattern IN expr + { Extensions.Comprehensions. + { pattern = $3 + ; iterator = In (mkexp_stack ~loc:$sloc (* ~kwd_loc:($loc($2)) *) $5) + ; attributes = $1 + } + } +; + +comprehension_clause: + | FOR separated_nonempty_llist(AND, comprehension_clause_binding) + { Extensions.Comprehensions.For $2 } + | WHEN expr + { Extensions.Comprehensions.When $2 } + +%inline comprehension(lbracket, rbracket): + lbracket expr nonempty_llist(comprehension_clause) rbracket + { Extensions.Comprehensions.{ body = $2; clauses = $3 } } +; + +%inline comprehension_ext_expr: + | comprehension(LBRACKET,RBRACKET) + { Extensions.Comprehensions.Cexp_list_comprehension $1 } + | comprehension(LBRACKETBAR,BARRBRACKET) + { Extensions.Comprehensions.Cexp_array_comprehension + (Mutable Location.none, $1) } + | comprehension(LBRACKETCOLON,COLONRBRACKET) + { Extensions.Comprehensions.Cexp_array_comprehension (Immutable, $1) } +; + +%inline comprehension_expr: + comprehension_ext_expr + { (Extensions.Comprehensions.expr_of ~loc:(make_loc $sloc) $1).pexp_desc } +; + +%inline array_simple(ARR_OPEN, ARR_CLOSE, contents_semi_list): + | ARR_OPEN contents_semi_list ARR_CLOSE + { Generic_array.Literal $2 } + | ARR_OPEN contents_semi_list error + { Generic_array.Unclosed($loc($1),$loc($3)) } + | ARR_OPEN ARR_CLOSE + { Generic_array.Literal [] } +; + +%inline array_exprs(ARR_OPEN, ARR_CLOSE): + | array_simple(ARR_OPEN, ARR_CLOSE, expr_semi_list) + { $1 } + | od=open_dot_declaration DOT ARR_OPEN expr_semi_list ARR_CLOSE + { Generic_array.Opened_literal(od, $startpos($3), $endpos, $4) } + | od=open_dot_declaration DOT ARR_OPEN ARR_CLOSE + { (* TODO: review the location of Pexp_array *) + Generic_array.Opened_literal(od, $startpos($3), $endpos, []) } + | mod_longident DOT + ARR_OPEN expr_semi_list error + { Generic_array.Unclosed($loc($3), $loc($5)) } +; + +%inline array_patterns(ARR_OPEN, ARR_CLOSE): + | array_simple(ARR_OPEN, ARR_CLOSE, pattern_semi_list) + { $1 } +; + +(* Jane Street extension *) +%inline hash: + | HASH { () } + | HASH_SUFFIX { () } +; +(* End Jane Street extension *) + %inline simple_expr_: | mkrhs(val_longident) { Pexp_ident ($1) } @@ -2393,7 +2699,7 @@ simple_expr: Pexp_open(od, mkexp ~loc:$sloc (Pexp_override $4)) } | mod_longident DOT LBRACELESS object_expr_content error { unclosed "{<" $loc($3) ">}" $loc($5) } - | simple_expr HASH mkrhs(label) + | simple_expr hash mkrhs(label) { Pexp_send($1, $3) } | simple_expr op(HASHOP) simple_expr { mkinfix $1 $2 $3 } @@ -2416,24 +2722,26 @@ simple_expr: (Pexp_record(fields, exten))) } | mod_longident DOT LBRACE record_expr_content error { unclosed "{" $loc($3) "}" $loc($5) } - | LBRACKETBAR expr_semi_list BARRBRACKET - { Pexp_array($2) } - | LBRACKETBAR expr_semi_list error - { unclosed "[|" $loc($1) "|]" $loc($3) } - | LBRACKETBAR BARRBRACKET - { Pexp_array [] } - | od=open_dot_declaration DOT LBRACKETBAR expr_semi_list BARRBRACKET - { Pexp_open(od, mkexp ~loc:($startpos($3), $endpos) (Pexp_array($4))) } - | od=open_dot_declaration DOT LBRACKETBAR BARRBRACKET - { (* TODO: review the location of Pexp_array *) - Pexp_open(od, mkexp ~loc:($startpos($3), $endpos) (Pexp_array [])) } - | mod_longident DOT - LBRACKETBAR expr_semi_list error - { unclosed "[|" $loc($3) "|]" $loc($5) } + | array_exprs(LBRACKETBAR, BARRBRACKET) + { Generic_array.expression + "[|" "|]" + (fun elts -> Pexp_array elts) + $1 } + | array_exprs(LBRACKETCOLON, COLONRBRACKET) + { Generic_array.expression + "[:" ":]" + (fun elts -> + (Extensions.Immutable_arrays.expr_of + ~loc:(make_loc $sloc) + (Iaexp_immutable_array elts)).pexp_desc) + $1 } | LBRACKET expr_semi_list RBRACKET { Pexp_list $2 } | LBRACKET expr_semi_list error { unclosed "[" $loc($1) "]" $loc($3) } + | comprehension_expr { $1 } + | od=open_dot_declaration DOT comprehension_expr + { Pexp_open(od, mkexp ~loc:($loc($3)) $3) } | od=open_dot_declaration DOT LBRACKET expr_semi_list RBRACKET { let list_exp = mkexp ~loc:($startpos($3), $endpos) (Pexp_list $4) in Pexp_open(od, list_exp) } @@ -2480,21 +2788,25 @@ labeled_simple_expr: let_binding_body_no_punning: let_ident strict_binding { ($1, $2, None) } - | let_ident type_constraint EQUAL seq_expr - { let v = $1 in (* PR#7344 *) + | optional_local let_ident type_constraint EQUAL seq_expr + { let v = $2 in (* PR#7344 *) let t = - match $2 with + match $3 with Some t, None -> Pvc_constraint { locally_abstract_univars = []; typ=t } | ground, Some coercion -> Pvc_coercion { ground; coercion} | _ -> assert false in - (v, $4, Some t) + let pat = mkpat_local_if $1 v in + let exp = mkexp_local_if $1 ~loc:$sloc $5 in + (pat, exp, Some t) } - | let_ident COLON poly(core_type) EQUAL seq_expr + | optional_local let_ident COLON poly(core_type) EQUAL seq_expr { - let t = ghtyp ~loc:($loc($3)) $3 in - ($1, $5, Some (Pvc_constraint { locally_abstract_univars = []; typ=t })) + let t = ghtyp ~loc:($loc($4)) $4 in + let pat = mkpat_local_if $1 $2 in + let exp = mkexp_local_if $1 ~loc:$sloc $6 in + (pat, exp, Some (Pvc_constraint { locally_abstract_univars = []; typ=t })) } | let_ident COLON TYPE lident_list DOT core_type EQUAL seq_expr { let constraint' = @@ -2505,6 +2817,8 @@ let_binding_body_no_punning: { ($1, $3, None) } | simple_pattern_not_ident COLON core_type EQUAL seq_expr { ($1, $5, Some(Pvc_constraint { locally_abstract_univars=[]; typ=$3 })) } + | LOCAL let_ident local_strict_binding + { ($2, mkexp_stack ~loc:$sloc $3, None) } ; let_binding_body: | let_binding_body_no_punning @@ -2583,6 +2897,20 @@ strict_binding: | LPAREN TYPE lident_list RPAREN fun_binding { mk_newtypes ~loc:$sloc $3 $5 } ; +local_fun_binding: + local_strict_binding + { $1 } + | type_constraint EQUAL seq_expr + { wrap_exp_stack (mkexp_constraint ~loc:$sloc $3 $1) } +; +local_strict_binding: + EQUAL seq_expr + { $2 } + | labeled_simple_pattern local_fun_binding + { let (l, o, p) = $1 in ghexp ~loc:$sloc (Pexp_fun(l, o, p, $2)) } + | LPAREN TYPE lident_list RPAREN local_fun_binding + { mk_newtypes ~loc:$sloc $3 $5 } +; %inline match_cases: xs = preceded_or_separated_nonempty_llist(BAR, match_case) { xs } @@ -2763,7 +3091,7 @@ simple_pattern_not_ident: { Ppat_construct($1, None) } | name_tag { Ppat_variant($1, None) } - | HASH mkrhs(type_longident) + | hash mkrhs(type_longident) { Ppat_type ($2) } | mkrhs(mod_longident) DOT simple_delimited_pattern { Ppat_open($1, $3) } @@ -2803,12 +3131,16 @@ simple_delimited_pattern: { Ppat_list $2 } | LBRACKET pattern_semi_list error { unclosed "[" $loc($1) "]" $loc($3) } - | LBRACKETBAR pattern_semi_list BARRBRACKET - { Ppat_array $2 } - | LBRACKETBAR BARRBRACKET - { Ppat_array [] } - | LBRACKETBAR pattern_semi_list error - { unclosed "[|" $loc($1) "|]" $loc($3) } + | array_patterns(LBRACKETBAR, BARRBRACKET) + { Generic_array.pattern + "[|" "|]" + (fun elts -> Ppat_array elts) + $1 } + | array_patterns(LBRACKETCOLON, COLONRBRACKET) + { Generic_array.pattern + "[:" ":]" + (ppat_iarray $sloc) + $1 } ) { $1 } pattern_comma_list(self): @@ -2991,12 +3323,27 @@ type_parameters: { [] } | p = type_parameter { [p] } - | LPAREN ps = separated_nonempty_llist(COMMA, type_parameter) RPAREN + | LPAREN + ps = separated_nonempty_llist(COMMA, parenthesized_type_parameter) + RPAREN { ps } ; + +layout: + ident { check_layout $loc($1) $1 } +; + +parenthesized_type_parameter: + type_parameter { $1 } + | type_variance type_variable COLON layout + { {$2 with ptyp_attributes = [$4]}, $1 } +; + type_parameter: - type_variance type_variable { $2, $1 } + type_variance type_variable attributes + { {$2 with ptyp_attributes = $3}, $1 } ; + type_variable: mktyp( QUOTE tyvar = ident @@ -3109,8 +3456,14 @@ generalized_constructor_arguments: { ($2,Pcstr_tuple [],Some $4) } ; +%inline atomic_type_gbl: + gbl = global_flag cty = atomic_type { + mkcty_global_maybe gbl cty (make_loc $loc(gbl)) +} +; + constructor_arguments: - | tys = inline_separated_nonempty_llist(STAR, atomic_type) + | tys = inline_separated_nonempty_llist(STAR, atomic_type_gbl) %prec below_HASH { Pcstr_tuple tys } | LBRACE label_declarations RBRACE @@ -3122,18 +3475,25 @@ label_declarations: | label_declaration_semi label_declarations { $1 :: $2 } ; label_declaration: - mutable_flag mkrhs(label) COLON poly_type_no_attr attributes + mutable_or_global_flag mkrhs(label) COLON poly_type_no_attr attributes { let info = symbol_info $endpos in - Type.field $2 $4 ~mut:$1 ~attrs:$5 ~loc:(make_loc $sloc) ~info } + let mut, gbl = $1 in + mkld_global_maybe gbl + (Type.field $2 $4 ~mut ~attrs:$5 ~loc:(make_loc $sloc) ~info) + (make_loc $loc($1)) } ; label_declaration_semi: - mutable_flag mkrhs(label) COLON poly_type_no_attr attributes SEMI attributes + mutable_or_global_flag mkrhs(label) COLON poly_type_no_attr attributes + SEMI attributes { let info = match rhs_info $endpos($5) with | Some _ as info_before_semi -> info_before_semi | None -> symbol_info $endpos in - Type.field $2 $4 ~mut:$1 ~attrs:($5 @ $7) ~loc:(make_loc $sloc) ~info } + let mut, gbl = $1 in + mkld_global_maybe gbl + (Type.field $2 $4 ~mut ~attrs:($5 @ $7) ~loc:(make_loc $sloc) ~info) + (make_loc $loc($1)) } ; /* Type Extensions */ @@ -3298,15 +3658,20 @@ function_type: | ty = tuple_type %prec MINUSGREATER { ty } + | ty = strict_function_type + { ty } +; +strict_function_type: | mktyp( label = arg_label - domain = extra_rhs(tuple_type) + local = optional_local + domain = extra_rhs(param_type) MINUSGREATER - codomain = function_type + codomain = strict_function_type { let arrow_type = { pap_label = label; pap_loc = make_loc $sloc; - pap_type = domain; + pap_type = mktyp_local_if local domain; } in let params, codomain = @@ -3318,6 +3683,36 @@ function_type: } ) { $1 } + | mktyp( + label = arg_label + arg_local = optional_local + domain = extra_rhs(param_type) + MINUSGREATER + ret_local = optional_local + codomain = tuple_type + %prec MINUSGREATER + { let arrow_type = { + pap_label = label; + pap_loc = make_loc $sloc; + pap_type = mktyp_local_if arg_local domain + } + in + let codomain = + mktyp_local_if ret_local (maybe_curry_typ codomain) + in + Ptyp_arrow([arrow_type], codomain) + } + ) + { $1 } +; +%inline param_type: + | mktyp( + LPAREN vars = typevar_list DOT ty = core_type RPAREN + { Ptyp_poly(vars, ty) } + ) + { $1 } + | ty = tuple_type + { ty } ; %inline arg_label: | label = optlabel @@ -3327,6 +3722,12 @@ function_type: | /* empty */ { Nolabel } ; +%inline optional_local: + | /* empty */ + { false } + | LOCAL + { true } +; (* Tuple types include: - atomic types (see below); - proper tuple types: int * int * int list @@ -3363,7 +3764,11 @@ atomic_type: { Ptyp_any } | tys = actual_type_parameters tid = mkrhs(type_longident) - { Ptyp_constr(tid, tys) } + HASH_SUFFIX + { Jane.ptyp_constr_unboxed tid tys } + | tys = actual_type_parameters + tid = mkrhs(type_longident) + { Ptyp_constr(tid, tys) } %prec below_HASH | LESS meth_list GREATER { let (f, c) = $2 in Ptyp_object (f, c) } | LESS GREATER @@ -3505,6 +3910,13 @@ constant: mkconst ~loc:$sloc (Pconst_string (s,strloc,d)) } | FLOAT { let (f, m) = $1 in mkconst ~loc:$sloc (Pconst_float (f, m)) } + + (* Jane Street extension *) + | HASH_INT { let (n, m) = $1 in + mkconst ~loc:$sloc (Jane.pconst_unboxed_integer Positive n m) } + | HASH_FLOAT { let (f, m) = $1 in + mkconst ~loc:$sloc (Jane.pconst_unboxed_float Positive f m) } + (* End Jane Street extension *) ; signed_constant: constant { $1 } @@ -3516,6 +3928,21 @@ signed_constant: mkconst ~loc:$sloc (Pconst_integer (n, m)) } | PLUS FLOAT { let (f, m) = $2 in mkconst ~loc:$sloc (Pconst_float(f, m)) } + + (* Jane Street extension *) + | MINUS HASH_INT { let (n, m) = $2 in + mkconst ~loc:$sloc + (Jane.pconst_unboxed_integer Negative n m) } + | MINUS HASH_FLOAT { let (f, m) = $2 in + mkconst ~loc:$sloc + (Jane.pconst_unboxed_float Negative f m) } + | PLUS HASH_INT { let (n, m) = $2 in + mkconst ~loc:$sloc + (Jane.pconst_unboxed_integer Positive n m) } + | PLUS HASH_FLOAT { let (f, m) = $2 in + mkconst ~loc:$sloc + (Jane.pconst_unboxed_float Positive f m) } + (* End Jane Street extension *) ; /* Identifiers and long identifiers */ @@ -3651,7 +4078,7 @@ any_longident: /* Toplevel directives */ toplevel_directive: - HASH dir = mkrhs(ident) + hash dir = mkrhs(ident) arg = ioption(mk_directive_arg(toplevel_directive_argument)) { mk_directive ~loc:$sloc dir arg } ; @@ -3712,6 +4139,16 @@ mutable_flag: /* empty */ { Immutable } | MUTABLE { Mutable (make_loc $sloc) } ; +mutable_or_global_flag: + /* empty */ { Immutable, Nothing } + | MUTABLE { Mutable (make_loc $sloc), + Nothing } + | GLOBAL { Immutable, Global } +; +%inline global_flag: + { Nothing } + | GLOBAL { Global } +; virtual_flag: /* empty */ { Concrete } | VIRTUAL { Virtual (make_loc $sloc) } @@ -3810,6 +4247,7 @@ single_attr_id: | INITIALIZER { "initializer" } | LAZY { "lazy" } | LET { "let" } + | LOCAL { "local_" } | MATCH { "match" } | METHOD { "method" } | MODULE { "module" } diff --git a/vendor/parser-extended/parsetree.mli b/vendor/parser-extended/parsetree.mli index dea619a38b..d168129884 100644 --- a/vendor/parser-extended/parsetree.mli +++ b/vendor/parser-extended/parsetree.mli @@ -22,6 +22,10 @@ open Asttypes +(* Jane Street extension *) +type sign = Positive | Negative +(* End Jane Street extension *) + type constant_desc = | Pconst_integer of string * char option (** Integer constants such as [3] [3l] [3L] [3n]. @@ -46,6 +50,17 @@ type constant_desc = Suffixes are rejected by the typechecker. *) + (* Jane street extension *) + (* Unboxed literals *) + | Pconst_unboxed_integer of sign * string * char option + (* [#3l], [-#42n] *) + (* The suffix is required, but ocamlformat need not enforce this, and + it's easier not to *) + + | Pconst_unboxed_float of sign * string * char option + (* [#3.0], [-#4.] *) + (* End Jane Street extension *) + type constant = { pconst_desc : constant_desc; pconst_loc : Location.t; @@ -189,6 +204,10 @@ and core_type_desc = | Ptyp_package of package_type (** [(module S)]. *) | Ptyp_extension of extension (** [[%id]]. *) + (* Jane Street extension *) + | Ptyp_constr_unboxed of Longident.t loc * core_type list + (* End Jane Street extension *) + and package_type = Longident.t loc * (Longident.t loc * core_type) list (** As {!package_type} typed values: - [(S, [])] represents [(module S)], @@ -876,6 +895,7 @@ and module_type_desc = | Pmty_typeof of module_expr (** [module type of ME] *) | Pmty_extension of extension (** [[%id]] *) | Pmty_alias of Longident.t loc (** [(module M)] *) + | Pmty_strengthen of module_type * Longident.t loc and functor_parameter = | Unit (** [()] *) diff --git a/vendor/parser-extended/printast.ml b/vendor/parser-extended/printast.ml index 51f196eecb..cfe98567d1 100644 --- a/vendor/parser-extended/printast.ml +++ b/vendor/parser-extended/printast.ml @@ -93,6 +93,12 @@ let fmt_char_option f = function | None -> fprintf f "None" | Some c -> fprintf f "Some %c" c +(* Jane Street extension *) +let fmt_sign f = function + | Positive -> fprintf f "Positive" + | Negative -> fprintf f "Negative" +(* End Jane Street extension *) + let fmt_constant i f x = line i f "constant %a\n" fmt_location x.pconst_loc; let i = i+1 in @@ -105,6 +111,13 @@ let fmt_constant i f x = line i f "PConst_string (%S,%a,Some %S)\n" s fmt_location strloc delim | Pconst_float (s,m) -> line i f "PConst_float (%s,%a)\n" s fmt_char_option m + (* Jane Street extension *) + | Pconst_unboxed_integer (s,j,m) -> + line i f "PConst_unboxed_integer (%a,%s,%a)" fmt_sign s j fmt_char_option m + | Pconst_unboxed_float (s,j,m) -> + line i f "PConst_unboxed_float (%a,%s,%a)" fmt_sign s j fmt_char_option m + (* End Jane Street extension *) + let fmt_mutable_flag f x = match x with | Immutable -> fprintf f "Immutable" @@ -235,6 +248,12 @@ let rec core_type i ppf x = line i ppf "Ptyp_extension %a\n" fmt_string_loc s; payload i ppf arg + (* Jane Street extension *) + | Ptyp_constr_unboxed (li, l) -> + line i ppf "Ptyp_constr_unboxed %a\n" fmt_longident_loc li; + list i core_type ppf l + (* End Jane Street extension *) + and arrow_param i ppf {pap_label; pap_loc; pap_type} = line i ppf "arrow_param %a\n" fmt_location pap_loc; arg_label i ppf pap_label; @@ -824,6 +843,10 @@ and module_type i ppf x = | Pmty_extension (s, arg) -> line i ppf "Pmod_extension %a\n" fmt_string_loc s; payload i ppf arg + | Pmty_strengthen (mt, li) -> + line i ppf "Pmty_strengthen\n"; + module_type i ppf mt; + longident_loc i ppf li and signature i ppf x = list i signature_item ppf x diff --git a/vendor/parser-shims/parser_shims.ml b/vendor/parser-shims/parser_shims.ml index 6f00d9be13..29bf498481 100644 --- a/vendor/parser-shims/parser_shims.ml +++ b/vendor/parser-shims/parser_shims.ml @@ -31,6 +31,8 @@ module Misc = struct let default_setting = Contextual end + + type (_, _) eq = Refl : ('a, 'a) eq end module Clflags = struct diff --git a/vendor/parser-shims/parser_shims.mli b/vendor/parser-shims/parser_shims.mli index a644b9fb5f..142fdca2c6 100644 --- a/vendor/parser-shims/parser_shims.mli +++ b/vendor/parser-shims/parser_shims.mli @@ -1,12 +1,12 @@ module List : sig - include module type of List + include module type of struct include List end val find_map : ('a -> 'b option) -> 'a list -> 'b option (** @since ocaml-4.10 *) end module Int : sig - include module type of Int + include module type of struct include Int end val min : int -> int -> int (** @since ocaml-4.13.0 *) @@ -16,21 +16,24 @@ module Int : sig end module Misc : sig - include module type of Misc + include module type of struct include Misc end module Color : sig - include module type of Color + include module type of struct include Color end val default_setting : setting (** @since ocaml-4.09 *) end module Error_style : sig - include module type of Error_style + include module type of struct include Error_style end val default_setting : setting (** @since ocaml-4.09 *) end + + (** Propositional equality *) + type (_, _) eq = Refl : ('a, 'a) eq end module Clflags : sig diff --git a/vendor/parser-standard/asttypes.mli b/vendor/parser-standard/asttypes.mli index 7a4f1c1913..7a531516ca 100644 --- a/vendor/parser-standard/asttypes.mli +++ b/vendor/parser-standard/asttypes.mli @@ -44,6 +44,19 @@ type override_flag = Override | Fresh type closed_flag = Closed | Open +type global_flag = + | Global + | Nothing + +(* constant layouts are parsed as layout annotations, and also used + in the type checker as already-inferred (i.e. non-variable) layouts *) +type const_layout = + | Any + | Value + | Void + | Immediate64 + | Immediate + type label = string type arg_label = diff --git a/vendor/parser-standard/jane_syntax.ml b/vendor/parser-standard/jane_syntax.ml new file mode 100644 index 0000000000..840bc09e68 --- /dev/null +++ b/vendor/parser-standard/jane_syntax.ml @@ -0,0 +1,639 @@ +open Asttypes +open Parsetree +open Jane_syntax_parsing + +(******************************************************************************) +(** Individual language extension modules *) + +(* Note [Check for immutable extension in comprehensions code] + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + + When we spot a comprehension for an immutable array, we need to make sure + that both [comprehensions] and [immutable_arrays] are enabled. But our + general mechanism for checking for enabled extensions (in [of_ast]) won't + work well here: it triggers when converting from + e.g. [[%jane.non_erasable.comprehensions.array] ...] to the + comprehensions-specific AST. But if we spot a + [[%jane.non_erasable.comprehensions.immutable]], there is no expression to + translate. So we just check for the immutable arrays extension when + processing a comprehension expression for an immutable array. + + Note [Wrapping with make_entire_jane_syntax] + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + + The topmost node in the encoded AST must always look like e.g. + [%jane.non_erasable.comprehensions]. (More generally, + [%jane.ERASABILITY.FEATURE] or [@jane.ERASABILITY.FEATURE].) This allows the + decoding machinery to know what extension is being used and what function to + call to do the decoding. Accordingly, during encoding, after doing the hard + work of converting the extension syntax tree into e.g. Parsetree.expression, + we need to make a final step of wrapping the result in a [%jane.*.xyz] node. + Ideally, this step would be done by part of our general structure, like we + separate [of_ast] and [of_ast_internal] in the decode structure; this design + would make it structurally impossible/hard to forget taking this final step. + + However, the final step is only one line of code (a call to + [make_entire_jane_syntax]), but yet the name of the feature varies, as does + the type of the payload. It would thus take several lines of code to execute + this command otherwise, along with dozens of lines to create the structure in + the first place. And so instead we just manually call + [make_entire_jane_syntax] and refer to this Note as a reminder to authors of + future syntax features to remember to do this wrapping. +*) + +module Builtin = struct + let make_curry_attr, extract_curry_attr, has_curry_attr = + Embedded_name.marker_attribute_handler ["curry"] + + let is_curried typ = has_curry_attr typ.ptyp_attributes + + let mark_curried ~loc typ = match typ.ptyp_desc with + | Ptyp_arrow _ when not (is_curried typ) -> + Core_type.add_attributes [make_curry_attr ~loc] typ + | _ -> typ + + let non_syntax_attributes attrs = + Option.value ~default:attrs (extract_curry_attr attrs) +end + +(** Locality modes *) +module Local = struct + let feature : Feature.t = Language_extension Local + + type constructor_argument = Lcarg_global of core_type + + type nonrec core_type = Ltyp_local of core_type + + type nonrec expression = + | Lexp_local of expression + | Lexp_exclave of expression + | Lexp_constrain_local of expression + (* Invariant: [Lexp_constrain_local] is the direct child of a + [Pexp_constraint] or [Pexp_coerce] node. For more, see the [.mli] + file. *) + + type nonrec pattern = Lpat_local of pattern + (* Invariant: [Lpat_local] is always the outermost part of a pattern. *) + + let type_of ~loc ~attrs = function + | Ltyp_local typ -> + (* See Note [Wrapping with make_entire_jane_syntax] *) + Core_type.make_entire_jane_syntax ~loc feature (fun () -> + (* Although there's only one constructor here, the use of + [constructor_argument] means we need to be able to tell the two uses + apart *) + Core_type.make_jane_syntax feature ["type"; "local"] @@ + Core_type.add_attributes attrs typ) + + let of_type = Core_type.match_jane_syntax_piece feature @@ fun typ -> function + | ["type"; "local"] -> Some (Ltyp_local typ) + | _ -> None + + let constr_arg_of ~loc lcarg = + (* See Note [Wrapping with make_entire_jane_syntax] *) + Constructor_argument.make_entire_jane_syntax ~loc feature (fun () -> + match lcarg with + | Lcarg_global carg -> + (* Although there's only one constructor here, the use of [core_type] + means we need to be able to tell the two uses apart *) + Constructor_argument.make_jane_syntax + feature ["constructor_argument"; "global"] + carg) + + let of_constr_arg = + Constructor_argument.match_jane_syntax_piece feature @@ fun carg -> function + | ["constructor_argument"; "global"] -> Some (Lcarg_global carg) + | _ -> None + + let expr_of ~loc ~attrs = function + | Lexp_local expr -> + (* See Note [Wrapping with make_entire_jane_syntax] *) + Expression.make_entire_jane_syntax ~loc feature (fun () -> + Expression.make_jane_syntax feature ["local"] @@ + Expression.add_attributes attrs expr) + | Lexp_exclave expr -> + (* See Note [Wrapping with make_entire_jane_syntax] *) + Expression.make_entire_jane_syntax ~loc feature (fun () -> + Expression.make_jane_syntax feature ["exclave"] @@ + Expression.add_attributes attrs expr) + | Lexp_constrain_local expr -> + (* See Note [Wrapping with make_entire_jane_syntax] *) + Expression.make_entire_jane_syntax ~loc feature (fun () -> + Expression.make_jane_syntax feature ["constrain_local"] @@ + Expression.add_attributes attrs expr) + + let of_expr = + Expression.match_jane_syntax_piece feature @@ fun expr -> function + | ["local"] -> Some (Lexp_local expr) + | ["exclave"] -> Some (Lexp_exclave expr) + | ["constrain_local"] -> Some (Lexp_constrain_local expr) + | _ -> None + + let pat_of ~loc ~attrs = function + | Lpat_local pat -> + (* See Note [Wrapping with make_entire_jane_syntax] *) + Pattern.make_entire_jane_syntax ~loc feature (fun () -> + Pattern.add_attributes attrs pat) + + let of_pat pat = Lpat_local pat +end + +(** List and array comprehensions *) +module Comprehensions = struct + let feature : Feature.t = Language_extension Comprehensions + + type iterator = + | Range of { start : expression + ; stop : expression + ; direction : direction_flag } + | In of expression + + type clause_binding = + { pattern : pattern + ; iterator : iterator + ; attributes : attribute list } + + type clause = + | For of clause_binding list + | When of expression + + type comprehension = + { body : expression + ; clauses : clause list + } + + type expression = + | Cexp_list_comprehension of comprehension + | Cexp_array_comprehension of mutable_flag * comprehension + + (* The desugared-to-OCaml version of comprehensions is described by the + following BNF, where [{% '...' | expr %}] refers to the result of + [Expression.make_jane_syntax] (via [comprehension_expr]) as described at + the top of [jane_syntax_parsing.mli]. + + {v + comprehension ::= + | {% 'comprehension.list' | '[' clauses ']' %} + | {% 'comprehension.array' | '[|' clauses '|]' %} + + clauses ::= + | {% 'comprehension.for' | 'let' iterator+ 'in' clauses %} + | {% 'comprehension.when' | expr ';' clauses %} + | {% 'comprehension.body' | expr %} + + iterator ::= + | pattern '=' {% 'comprehension.for.range.upto' | expr ',' expr %} + | pattern '=' {% 'comprehension.for.range.downto' | expr ',' expr %} + | pattern '=' {% 'comprehension.for.in' | expr %} + v} + *) + + let comprehension_expr = Expression.make_jane_syntax feature + + (** First, we define how to go from the nice AST to the OCaml AST; this is + the [expr_of_...] family of expressions, culminating in [expr_of]. *) + + let expr_of_iterator = function + | Range { start; stop; direction } -> + comprehension_expr + [ "for" + ; "range" + ; match direction with + | Upto -> "upto" + | Downto -> "downto" ] + (Ast_helper.Exp.tuple [start; stop]) + | In seq -> + comprehension_expr ["for"; "in"] (Ast_helper.Exp.lazy_ seq) + (* See Note [Wrapping with Pexp_lazy] *) + + let expr_of_clause_binding { pattern; iterator; attributes } = + Ast_helper.Vb.mk ~attrs:attributes pattern (expr_of_iterator iterator) + + let expr_of_clause clause rest = match clause with + | For iterators -> + comprehension_expr + ["for"] + (Ast_helper.Exp.let_ + Nonrecursive (List.map expr_of_clause_binding iterators) + rest) + | When cond -> + comprehension_expr ["when"] (Ast_helper.Exp.sequence cond rest) + + let expr_of_comprehension ~type_ ~attrs { body; clauses } = + (* See Note [Wrapping with Pexp_lazy] *) + comprehension_expr + type_ + (Expression.add_attributes + attrs + (Ast_helper.Exp.lazy_ + (List.fold_right + expr_of_clause + clauses + (comprehension_expr ["body"] (Ast_helper.Exp.lazy_ body))))) + + let expr_of ~loc ~attrs cexpr = + (* See Note [Wrapping with make_entire_jane_syntax] *) + Expression.make_entire_jane_syntax ~loc feature (fun () -> + match cexpr with + | Cexp_list_comprehension comp -> + expr_of_comprehension ~type_:["list"] ~attrs comp + | Cexp_array_comprehension (amut, comp) -> + expr_of_comprehension + ~type_:[ "array" + ; match amut with + | Mutable -> "mutable" + | Immutable -> "immutable" + ] + ~attrs + comp) + + (** Then, we define how to go from the OCaml AST to the nice AST; this is + the [..._of_expr] family of expressions, culminating in [of_expr]. *) + + module Desugaring_error = struct + type error = + | No_clauses + | Unexpected_attributes of attributes + (* Note [Wrapping with Pexp_lazy] + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + + We require that every internal comprehensions node contain at least one + constructor, using [Pexp_lazy] by convention when there isn't another + obvious choice. This means that every internal AST node synthesized + for comprehensions can contain no other attributes, which we can then + check for and raise [Unexpected_attributes] if we get this wrong. This + helps guard against attribute erros. *) + + let report_error ~loc = function + | No_clauses -> + Location.errorf ~loc + "Tried to desugar a comprehension with no clauses" + | Unexpected_attributes attrs -> + Location.errorf ~loc + "An internal synthesized comprehension node had extra attributes.@.\ + The attributes had the following names:@ %a" + (Format.pp_print_list + ~pp_sep:(fun ppf () -> Format.fprintf ppf ",@ ") + (fun ppf attr -> Format.fprintf ppf "\"%s\"" attr.attr_name.txt)) + attrs + + exception Error of Location.t * error + + let () = + Location.register_error_of_exn + (function + | Error(loc, err) -> Some (report_error ~loc err) + | _ -> None) + + let raise expr err = raise (Error(expr.pexp_loc, err)) + end + + let match_comprehension_piece matcher = + Expression.match_jane_syntax_piece feature @@ fun expr subparts -> + match expr.pexp_attributes with + | [] -> matcher expr subparts + | _ :: _ as attrs -> + Desugaring_error.raise expr (Unexpected_attributes attrs) + + let iterator_of_expr = match_comprehension_piece @@ fun expr subparts -> + match subparts, expr.pexp_desc with + |["for"; "range"; "upto"], Pexp_tuple [start; stop] -> + Some (Range { start; stop; direction = Upto }) + | ["for"; "range"; "downto"], Pexp_tuple [start; stop] -> + Some (Range { start; stop; direction = Downto }) + | ["for"; "in"], Pexp_lazy seq -> + Some (In seq) + | _ -> None + + let clause_binding_of_vb { pvb_pat; pvb_expr; pvb_attributes; pvb_loc = _ } = + { pattern = pvb_pat + ; iterator = iterator_of_expr pvb_expr + ; attributes = pvb_attributes } + + let add_clause clause comp = { comp with clauses = clause :: comp.clauses } + + let comprehension_of_expr = + let rec raw_comprehension_of_expr expr = + expr |> match_comprehension_piece @@ fun expr subparts -> + match subparts, expr.pexp_desc with + | ["for"], Pexp_let(Nonrecursive, iterators, rest) -> + Option.some @@ add_clause + (For (List.map clause_binding_of_vb iterators)) + (raw_comprehension_of_expr rest) + | ["when"], Pexp_sequence(cond, rest) -> + Option.some @@ add_clause + (When cond) + (raw_comprehension_of_expr rest) + | ["body"], Pexp_lazy body -> + Some { body; clauses = [] } + | _ -> + None + in + fun expr -> + match raw_comprehension_of_expr expr with + | { body = _; clauses = [] } -> + Desugaring_error.raise expr No_clauses + | comp -> comp + + let of_expr = match_comprehension_piece @@ fun expr subparts -> + (* See Note [Wrapping with Pexp_lazy] *) + match subparts, expr.pexp_desc with + | ["list"], Pexp_lazy comp -> + Some (Cexp_list_comprehension (comprehension_of_expr comp)) + | ["array"; "mutable"], Pexp_lazy comp -> + Some (Cexp_array_comprehension (Mutable, + comprehension_of_expr comp)) + | ["array"; "immutable"], Pexp_lazy comp -> + (* assert_extension_enabled: + See Note [Check for immutable extension in comprehensions code] + *) + assert_extension_enabled ~loc:expr.pexp_loc Immutable_arrays (); + Some (Cexp_array_comprehension (Immutable, + comprehension_of_expr comp)) + | _ -> None +end + +(** Immutable arrays *) +module Immutable_arrays = struct + type nonrec expression = + | Iaexp_immutable_array of expression list + + type nonrec pattern = + | Iapat_immutable_array of pattern list + + let feature : Feature.t = Language_extension Immutable_arrays + + let expr_of ~loc ~attrs = function + | Iaexp_immutable_array elts -> + (* See Note [Wrapping with make_entire_jane_syntax] *) + Expression.make_entire_jane_syntax ~loc feature (fun () -> + Ast_helper.Exp.array ~attrs elts) + + let of_expr expr = match expr.pexp_desc with + | Pexp_array elts -> Iaexp_immutable_array elts + | _ -> failwith "Malformed immutable array expression" + + let pat_of ~loc ~attrs = function + | Iapat_immutable_array elts -> + (* See Note [Wrapping with make_entire_jane_syntax] *) + Pattern.make_entire_jane_syntax ~loc feature (fun () -> + Ast_helper.Pat.array ~attrs elts) + + let of_pat pat = match pat.ppat_desc with + | Ppat_array elts -> Iapat_immutable_array elts + | _ -> failwith "Malformed immutable array pattern" +end + +(** [include functor] *) +module Include_functor = struct + type signature_item = + | Ifsig_include_functor of include_description + + type structure_item = + | Ifstr_include_functor of include_declaration + + let feature : Feature.t = Language_extension Include_functor + + let sig_item_of ~loc = function + | Ifsig_include_functor incl -> + (* See Note [Wrapping with make_entire_jane_syntax] *) + Signature_item.make_entire_jane_syntax ~loc feature (fun () -> + Ast_helper.Sig.include_ incl) + + let of_sig_item sigi = match sigi.psig_desc with + | Psig_include incl -> Ifsig_include_functor incl + | _ -> failwith "Malformed [include functor] in signature" + + let str_item_of ~loc = function + | Ifstr_include_functor incl -> + (* See Note [Wrapping with make_entire_jane_syntax] *) + Structure_item.make_entire_jane_syntax ~loc feature (fun () -> + Ast_helper.Str.include_ incl) + + let of_str_item stri = match stri.pstr_desc with + | Pstr_include incl -> Ifstr_include_functor incl + | _ -> failwith "Malformed [include functor] in structure" +end + +(** Module strengthening *) +module Strengthen = struct + type nonrec module_type = + { mty : Parsetree.module_type; mod_id : Longident.t Location.loc } + + let feature : Feature.t = Language_extension Module_strengthening + + (* Encoding: [S with M] becomes [functor (_ : S) -> (module M)], where + the [(module M)] is a [Pmty_alias]. This isn't syntax we can write, but + [(module M)] can be the inferred type for [M], so this should be fine. *) + + let mty_of ~loc ~attrs { mty; mod_id } = + (* See Note [Wrapping with make_entire_jane_syntax] *) + Module_type.make_entire_jane_syntax ~loc feature (fun () -> + Ast_helper.Mty.functor_ ~attrs (Named (Location.mknoloc None, mty)) + (Ast_helper.Mty.alias mod_id)) + + (* Returns remaining unconsumed attributes *) + let of_mty mty = match mty.pmty_desc with + | Pmty_functor(Named(_, mty), {pmty_desc = Pmty_alias mod_id}) -> + { mty; mod_id } + | _ -> failwith "Malformed strengthened module type" +end + +module Unboxed_constants = struct + type t = + | Float of string * char option + | Integer of string * char + + type expression = t + type pattern = t + + let feature : Feature.t = Language_extension Layouts + + let fail_malformed ~loc = + Location.raise_errorf ~loc "Malformed unboxed numeric literal" + + let of_constant ~loc = function + | Pconst_float (x, suffix) -> Float (x, suffix) + | Pconst_integer (x, Some suffix) -> Integer (x, suffix) + | Pconst_integer (_, None) -> + Location.raise_errorf ~loc + "Malformed unboxed int literal: suffix required" + | _ -> fail_malformed ~loc + + + (* Returns remaining unconsumed attributes *) + let of_expr expr = + let loc = expr.pexp_loc in + match expr.pexp_desc with + | Pexp_constant const -> of_constant ~loc const + | _ -> fail_malformed ~loc + + (* Returns remaining unconsumed attributes *) + let of_pat pat = + let loc = pat.ppat_loc in + match pat.ppat_desc with + | Ppat_constant const -> of_constant ~loc const + | _ -> fail_malformed ~loc + + let constant_of = function + | Float (x, suffix) -> Pconst_float (x, suffix) + | Integer (x, suffix) -> Pconst_integer (x, Some suffix) + + let expr_of ~loc ~attrs t = + let constant = constant_of t in + Expression.make_entire_jane_syntax ~loc feature (fun () -> + Ast_helper.Exp.constant ~attrs constant) + + let pat_of ~loc ~attrs t = + let constant = constant_of t in + Pattern.make_entire_jane_syntax ~loc feature (fun () -> + Ast_helper.Pat.constant ~attrs constant) +end + +(******************************************************************************) +(** The interface to our novel syntax, which we export *) + +module type AST = sig + type t + type ast + + val of_ast : ast -> t option + val ast_of : loc:Location.t -> t -> ast +end + +module Core_type = struct + type t = + | Jtyp_local of Local.core_type + + let of_ast_internal (feat : Feature.t) typ = match feat with + | Language_extension Local -> Some (Jtyp_local (Local.of_type typ)) + | _ -> None + + let of_ast = Core_type.make_of_ast ~of_ast_internal + + let ast_of ~loc (jtyp, attrs) = match jtyp with + | Jtyp_local x -> Local.type_of ~loc ~attrs x +end + +module Constructor_argument = struct + type t = + | Jcarg_local of Local.constructor_argument + + let of_ast_internal (feat : Feature.t) carg = match feat with + | Language_extension Local -> Some (Jcarg_local (Local.of_constr_arg carg)) + | _ -> None + + let of_ast = Constructor_argument.make_of_ast ~of_ast_internal + + let ast_of ~loc jcarg = match jcarg with + | Jcarg_local x -> Local.constr_arg_of ~loc x +end + +module Expression = struct + type t = + | Jexp_local of Local.expression + | Jexp_comprehension of Comprehensions.expression + | Jexp_immutable_array of Immutable_arrays.expression + | Jexp_unboxed_constant of Unboxed_constants.expression + + let of_ast_internal (feat : Feature.t) expr = match feat with + | Language_extension Local -> + Some (Jexp_local (Local.of_expr expr)) + | Language_extension Comprehensions -> + Some (Jexp_comprehension (Comprehensions.of_expr expr)) + | Language_extension Immutable_arrays -> + Some (Jexp_immutable_array (Immutable_arrays.of_expr expr)) + | Language_extension Layouts -> + Some (Jexp_unboxed_constant (Unboxed_constants.of_expr expr)) + | _ -> None + + let of_ast = Expression.make_of_ast ~of_ast_internal + + let ast_of ~loc (jexp, attrs) = match jexp with + | Jexp_local x -> Local.expr_of ~loc ~attrs x + | Jexp_comprehension x -> Comprehensions.expr_of ~loc ~attrs x + | Jexp_immutable_array x -> Immutable_arrays.expr_of ~loc ~attrs x + | Jexp_unboxed_constant x -> Unboxed_constants.expr_of ~loc ~attrs x +end + +module Pattern = struct + type t = + | Jpat_local of Local.pattern + | Jpat_immutable_array of Immutable_arrays.pattern + | Jpat_unboxed_constant of Unboxed_constants.pattern + + let of_ast_internal (feat : Feature.t) pat = match feat with + | Language_extension Local -> + Some (Jpat_local (Local.of_pat pat)) + | Language_extension Immutable_arrays -> + Some (Jpat_immutable_array (Immutable_arrays.of_pat pat)) + | Language_extension Layouts -> + Some (Jpat_unboxed_constant (Unboxed_constants.of_pat pat)) + | _ -> None + + let of_ast = Pattern.make_of_ast ~of_ast_internal + + let ast_of ~loc (jpat, attrs) = match jpat with + | Jpat_local x -> Local.pat_of ~loc ~attrs x + | Jpat_immutable_array x -> Immutable_arrays.pat_of ~loc ~attrs x + | Jpat_unboxed_constant x -> Unboxed_constants.pat_of ~loc ~attrs x +end + +module Module_type = struct + type t = + | Jmty_strengthen of Strengthen.module_type + + let of_ast_internal (feat : Feature.t) mty = match feat with + | Language_extension Module_strengthening -> + Some (Jmty_strengthen (Strengthen.of_mty mty)) + | _ -> None + + let of_ast = Module_type.make_of_ast ~of_ast_internal + + let ast_of ~loc (jmty, attrs) = match jmty with + | Jmty_strengthen x -> Strengthen.mty_of ~loc ~attrs x +end + +module Signature_item = struct + type t = + | Jsig_include_functor of Include_functor.signature_item + + let of_ast_internal (feat : Feature.t) sigi = + match feat with + | Language_extension Include_functor -> + Some (Jsig_include_functor (Include_functor.of_sig_item sigi)) + | _ -> None + + let of_ast = Signature_item.make_of_ast ~of_ast_internal + + let ast_of ~loc jsig = match jsig with + | Jsig_include_functor x -> Include_functor.sig_item_of ~loc x +end + +module Structure_item = struct + type t = + | Jstr_include_functor of Include_functor.structure_item + + let of_ast_internal (feat : Feature.t) stri = + match feat with + | Language_extension Include_functor -> + Some (Jstr_include_functor (Include_functor.of_str_item stri)) + | _ -> None + + let of_ast = Structure_item.make_of_ast ~of_ast_internal + + let ast_of ~loc jstr = match jstr with + | Jstr_include_functor x -> Include_functor.str_item_of ~loc x +end + +module Extension_constructor = struct + type t = | + + let of_ast_internal (feat : Feature.t) _ext = match feat with + | _ -> None + + let of_ast = Extension_constructor.make_of_ast ~of_ast_internal + + let ast_of ~loc:_ (jext, _attrs) = match jext with + | (_ : t) -> . +end diff --git a/vendor/parser-standard/jane_syntax.mli b/vendor/parser-standard/jane_syntax.mli new file mode 100644 index 0000000000..24fecb2ca9 --- /dev/null +++ b/vendor/parser-standard/jane_syntax.mli @@ -0,0 +1,389 @@ +(** Syntax for Jane Street's novel syntactic features. This module provides + three things: + + 1. First-class ASTs for all syntax introduced by our language extensions, + plus one for built-in features; these are split out into a different + module each ([Comprehensions], etc.). + + 2. A first-class AST for each OCaml AST, unifying all our novel syntactic + features in modules named after the syntactic category + ([Expression.t], etc.). + + 3. A way to interpret these values as terms of the coresponding OCaml ASTs, + and to match on terms of those OCaml ASTs to see if they're terms from + our novel syntax. + + We keep our novel syntax separate so that we can avoid having to modify the + existing AST, as this would break compatibility with every existing ppx and + other such tooling. + + For details on the rationale behind this approach (and for some of the gory + details), see [Jane_syntax_parsing]. *) + +(*********************************************) +(* Individual features *) + +(** The ASTs for built-in syntax extensions. No ASTs as yet; for now, we just + have some attribute machinery. *) +module Builtin : sig + (** Mark an arrow type as "curried" (written with parentheses) for the local + extension. This is done unconditionally by the parser: [a -> (b -> c)] is + parsed as [a -> ((b -> c)[@CURRY])] for some (private) attribute. A + non-arrow type won't be modified by this function. + + We leave this as an attribute because it's only used internally, and + changing function types/adding another kind of arrow is a *lot* of + work. *) + val mark_curried : + loc:Location.t -> Parsetree.core_type -> Parsetree.core_type + + (** Check if a type was marked as curried via [mark_curried]. Does not modify + the attributes of the type. *) + val is_curried : Parsetree.core_type -> bool + + (** Return all the attributes from the given list that were not added by + marking functions such as [mark_curried]. The same as accessing + [ptyp_attributes] if the type was not so marked. *) + val non_syntax_attributes : Parsetree.attributes -> Parsetree.attributes +end + +(** The ASTs for locality modes *) +module Local : sig + type core_type = Ltyp_local of Parsetree.core_type + (** [local_ TYPE] + + Invariant: Only used in arrow types (e.g., [local_ a -> local_ b]), and + has no attributes (the inner [core_type] can). + + The other part of locality that shows up in types is the marking of what's + curried (i.e., represented with explicit parentheses in the source); this + is represented by the [Builtin.mark_curried] machinery, which see. *) + + type constructor_argument = + | Lcarg_global of Parsetree.core_type + (** [global_ TYPE] + + E.g.: [type t = { x : global_ string }] or + [type t = C of global_ string]. *) + + type expression = + | Lexp_local of Parsetree.expression + (** [local_ EXPR] *) + | Lexp_exclave of Parsetree.expression + (** [exclave_ EXPR] *) + | Lexp_constrain_local of Parsetree.expression + (** This represents the shadow [local_] that is inserted on the RHS of a + [let local_ f : t = e in ...] binding. + + Invariant: [Lexp_constrain_local] occurs on the LHS of a + [Pexp_constraint] or [Pexp_coerce] node. + + We don't inline the definition of [Pexp_constraint] or [Pexp_coerce] + here because nroberts's (@ncik-roberts's) forthcoming syntactic + function arity parsing patch handles this case more directly, and we + don't want to double the amount of work we're doing. *) + + type pattern = + | Lpat_local of Parsetree.pattern + (** [local_ PAT] + + Invariant: [Lpat_local] is always the outermost part of a pattern. *) + + val type_of : + loc:Location.t -> attrs:Parsetree.attributes -> + core_type -> Parsetree.core_type + val constr_arg_of : + loc:Location.t -> constructor_argument -> Parsetree.core_type + val expr_of : + loc:Location.t -> attrs:Parsetree.attributes -> + expression -> Parsetree.expression + val pat_of : + loc:Location.t -> attrs:Parsetree.attributes -> + pattern -> Parsetree.pattern +end + +(** The ASTs for list and array comprehensions *) +module Comprehensions : sig + type iterator = + | Range of { start : Parsetree.expression + ; stop : Parsetree.expression + ; direction : Asttypes.direction_flag } + (** [= START to STOP] (direction = [Upto]) + [= START downto STOP] (direction = [Downto]) *) + | In of Parsetree.expression + (** [in EXPR] *) + + (* In [Typedtree], the [pattern] moves into the [iterator]. *) + type clause_binding = + { pattern : Parsetree.pattern + ; iterator : iterator + ; attributes : Parsetree.attribute list } + (** [[@...] PAT (in/=) ...] *) + + type clause = + | For of clause_binding list + (** [for PAT (in/=) ... and PAT (in/=) ... and ...]; must be nonempty *) + | When of Parsetree.expression + (** [when EXPR] *) + + type comprehension = + { body : Parsetree.expression + (** The body/generator of the comprehension *) + ; clauses : clause list + (** The clauses of the comprehension; must be nonempty *) } + + type expression = + | Cexp_list_comprehension of comprehension + (** [[BODY ...CLAUSES...]] *) + | Cexp_array_comprehension of Asttypes.mutable_flag * comprehension + (** [[|BODY ...CLAUSES...|]] (flag = [Mutable]) + [[:BODY ...CLAUSES...:]] (flag = [Immutable]) + (only allowed with [-extension immutable_arrays]) *) + + val expr_of : + loc:Location.t -> attrs:Parsetree.attributes -> + expression -> Parsetree.expression +end + +(** The ASTs for immutable arrays. When we merge this upstream, we'll merge + these into the existing [P{exp,pat}_array] constructors by adding a + [mutable_flag] argument (just as we did with [T{exp,pat}_array]). *) +module Immutable_arrays : sig + type expression = + | Iaexp_immutable_array of Parsetree.expression list + (** [[: E1; ...; En :]] *) + + type pattern = + | Iapat_immutable_array of Parsetree.pattern list + (** [[: P1; ...; Pn :]] **) + + val expr_of : + loc:Location.t -> attrs:Parsetree.attributes -> + expression -> Parsetree.expression + val pat_of : + loc:Location.t -> attrs:Parsetree.attributes -> + pattern -> Parsetree.pattern +end + +(** The ASTs for [include functor]. When we merge this upstream, we'll merge + these into the existing [P{sig,str}_include] constructors (similar to what + we did with [T{sig,str}_include], but without depending on typechecking). *) +module Include_functor : sig + type signature_item = + | Ifsig_include_functor of Parsetree.include_description + (** [include functor MTY] *) + + type structure_item = + | Ifstr_include_functor of Parsetree.include_declaration + (** [include functor MOD] *) + + val sig_item_of : loc:Location.t -> signature_item -> Parsetree.signature_item + val str_item_of : loc:Location.t -> structure_item -> Parsetree.structure_item +end + +(** The ASTs for module type strengthening. *) +module Strengthen : sig + type module_type = + { mty : Parsetree.module_type; mod_id : Longident.t Location.loc } + + val mty_of : + loc:Location.t -> attrs:Parsetree.attributes -> + module_type -> Parsetree.module_type +end + +(** The ASTs for unboxed literals, like #4.0 *) +module Unboxed_constants : sig + type t = + | Float of string * char option + (** Unboxed float constants such as [3.4#], [-2e5#], or [+1.4e-4#g]. + + Unlike with boxed constants, the sign (if present) is included. + + Suffixes [g-z][G-Z] are accepted by the parser. + Suffixes are rejected by the typechecker. *) + | Integer of string * char + (** Unboxed float constants such as [3#], [-3#l], [+3#L], or [3#n]. + + Unlike with boxed constants, the sign (if present) is included. + + Suffixes [g-z][G-Z] are *required* by the parser. + Suffixes except ['l'], ['L'] and ['n'] are rejected by the typechecker. + *) + + type expression = t + type pattern = t + + val expr_of : + loc:Location.t -> attrs:Parsetree.attributes -> + expression -> Parsetree.expression + + val pat_of : + loc:Location.t -> attrs:Parsetree.attributes -> + pattern -> Parsetree.pattern +end + +(******************************************) +(* General facility, which we export *) + +(** The module type of our extended ASTs for our novel syntax, instantiated once + for each syntactic category. We tend to call the pattern-matching functions + here with unusual indentation, not indenting the [None] branch further so as + to avoid merge conflicts with upstream. *) +module type AST = sig + (** The AST for all our Jane Street syntax; one constructor per feature that + extends the given syntactic category. Some extensions are handled + separately and thus are not listed here. + + This type will be something like [jane_syntax_ast * Parsetree.attributes] + in cases where the Jane Syntax encoding of the AST uses attributes. In + these cases, the [Parsetree.attributes] are the *rest* of the attributes + after removing Jane Syntax-related attributes. Callers of [of_ast] should + refer to these attributes rather than, for example, [pexp_attributes]. + *) + type t + + (** The corresponding OCaml AST *) + type ast + + (** Given an OCaml AST node, check to see if it corresponds to an embedded + term from our novel syntax. If it does, as long as the feature isn't a + disabled language extension, then return it; if it's not a piece of novel + syntax, return [None]; if it's an embedded term from a disabled language + extension, raise an error. + + AN IMPORTANT NOTE: The design of this function is careful to make merge + conflicts with upstream less likely: we want no edits at all -- not even + indentation -- to surrounding code. This is why we return a [t option], + not some structure that could include the [ast_desc] if there is no + extension. + + Indentation: we *do not change the indentation level* when we match on + this function's result! E.g. from [type_expect_] in [typecore.ml]: + + {[ + match Jane_syntax.Expression.of_ast sexp with + | Some jexp -> + type_expect_jane_syntax + ~loc + ~env + ~expected_mode + ~ty_expected + ~explanation + ~attributes:sexp.pexp_attributes + jexp + | None -> match sexp.pexp_desc with + | Pexp_ident lid -> + let path, mode, desc, kind = type_ident env ~recarg lid in + (* ... *) + | Pexp_constant(Pconst_string (str, _, _) as cst) -> + register_allocation expected_mode; + (* ... *) + | (* ... *) + | Pexp_unreachable -> + re { exp_desc = Texp_unreachable; + exp_loc = loc; exp_extra = []; + exp_type = instance ty_expected; + exp_mode = expected_mode.mode; + exp_attributes = sexp.pexp_attributes; + exp_env = env } + ]} + + Note that we match on the result of this function, forward to + [type_expect_jane_syntax] if we get something, and otherwise do the real + match on [sexp.pexp_desc] *without going up an indentation level*. This + is important to reduce the number of merge conflicts. *) + val of_ast : ast -> t option + + (** The dual of [of_ast], only used by [Ast_mapper]. This is built up from + the various [FEATURE.CATEGORY_of], such as [Local.type_of], which you + should prefer. This generic version allows for easier construction of + OCaml AST terms from Jane syntax ASTs when you don't know which Jane + syntax feature you have; this doesn't occur very frequently, hence the + limited use. *) + val ast_of : loc:Location.t -> t -> ast +end + +(******************************************) +(* Individual syntactic categories *) + +(** Novel syntax in types *) +module Core_type : sig + type t = + | Jtyp_local of Local.core_type + + include AST + with type t := t * Parsetree.attributes + and type ast := Parsetree.core_type +end + +(** Novel syntax in constructor arguments; this isn't a core AST type, but + captures where [global_] lives. Unlike types, they don't have attributes; + any attributes are either on the label declaration they're in (if any) or on + the inner type. *) +module Constructor_argument : sig + type t = + | Jcarg_local of Local.constructor_argument + + include AST + with type t := t + and type ast := Parsetree.core_type +end + +(** Novel syntax in expressions *) +module Expression : sig + type t = + | Jexp_local of Local.expression + | Jexp_comprehension of Comprehensions.expression + | Jexp_immutable_array of Immutable_arrays.expression + | Jexp_unboxed_constant of Unboxed_constants.expression + + include AST + with type t := t * Parsetree.attributes + and type ast := Parsetree.expression +end + +(** Novel syntax in patterns *) +module Pattern : sig + type t = + | Jpat_local of Local.pattern + | Jpat_immutable_array of Immutable_arrays.pattern + | Jpat_unboxed_constant of Unboxed_constants.pattern + + include AST + with type t := t * Parsetree.attributes + and type ast := Parsetree.pattern +end + +(** Novel syntax in module types *) +module Module_type : sig + type t = + | Jmty_strengthen of Strengthen.module_type + + include AST + with type t := t * Parsetree.attributes + and type ast := Parsetree.module_type +end + +(** Novel syntax in signature items *) +module Signature_item : sig + type t = + | Jsig_include_functor of Include_functor.signature_item + + include AST with type t := t and type ast := Parsetree.signature_item +end + +(** Novel syntax in structure items *) +module Structure_item : sig + type t = + | Jstr_include_functor of Include_functor.structure_item + + include AST with type t := t and type ast := Parsetree.structure_item +end + +(** Novel syntax in extension constructors *) +module Extension_constructor : sig + type t = | + + include AST with type t := t * Parsetree.attributes + and type ast := Parsetree.extension_constructor +end diff --git a/vendor/parser-standard/jane_syntax_parsing.ml b/vendor/parser-standard/jane_syntax_parsing.ml new file mode 100644 index 0000000000..923b5da6c9 --- /dev/null +++ b/vendor/parser-standard/jane_syntax_parsing.ml @@ -0,0 +1,1116 @@ +(** As mentioned in the .mli file, there are some gory details around the + particular translation scheme we adopt for moving to and from OCaml ASTs + ([Parsetree.expression], etc.). The general idea is that we adopt a scheme + where each novel piece of syntax is represented using one of two embeddings: + + 1. As an AST item carrying an attribute. The AST item serves as the "body" + of the syntax indicated by the attribute. + 2. As a pair of an extension node and an AST item that serves as the "body". + Here, the "pair" is embedded as a pair-like construct in the relevant AST + category, e.g. [include sig [%jane.ERASABILITY.EXTNAME];; BODY end] for + signature items. + + In particular, for an language extension named [EXTNAME] (i.e., one that is + enabled by [-extension EXTNAME] on the command line), the attribute (if + used) must be [[@jane.ERASABILITY.EXTNAME]], and the extension node (if + used) must be [[%jane.ERASABILITY.EXTNAME]]. For built-in syntax, we use + [_builtin] instead of a language extension name. + + The only exception to this is that for some built-in syntax, we instead use + certain "marker" attributes, designed to be created by the parser when a + full Jane-syntax encoding would be too heavyweight; for these, we use + [_marker] instead of an extension name, and allow arbitrary dot-separated + strings (see below) to follow it. + + The [ERASABILITY] component indicates to tools such as ocamlformat and + ppxlib whether or not the attribute is erasable. See the documentation of + [Erasability] for more information on how tools make use of this + information. + + In the below example, we use attributes an examples, but it applies equally + to extensions. We also provide utilities for further desugaring similar + applications where the embeddings have the longer form + [[@jane.ERASABILITY.FEATNAME.ID1.ID2.….IDn]] (with the outermost one being + the [n = 0] case), as these might be used inside the [EXPR]. (For example, + within the outermost [[@jane.non_erasable.comprehensions]] term for list and + array comprehensions, we can also use + [[@jane.non_erasable.comprehensions.list]], + [[@jane.non_erasable.comprehensions.array]], + [[@jane.non_erasable.comprehensions.for.in]], etc.). + + As mentioned, we represent terms as a "pair" and don't use the extension + node or attribute payload; this is so that ppxen can see inside these + extension nodes or attributes. If we put the subexpressions inside the + payload, then we couldn't write something like [[[%string "Hello, %{x}!"] + for x in names]], as [ppx_string] wouldn't traverse inside the payload to + find the [[%string]] extension node. + + Our novel syntactic features are of course allowed to impose extra + constraints on what legal bodies are; we're also happy for this translation + to error in various ways on malformed input, since nobody should ever be + writing these forms directly. They're just an implementation detail. + + See modules of type AST below to see how different syntactic categories + are represented. For example, expressions are encoded using an attribute. + + We provide one module per syntactic category (e.g., [Expression]), of module + type [AST]. They also provide some simple machinery for working with the + general [@jane.ERASABILITY.FEATNAME.ID1.ID2.….IDn] wrapped forms. To + construct one, we provide [make_jane_syntax]; to destructure one, we provide + [match_jane_syntax] (which we expose via [make_of_ast]). Users of this + module still have to write the transformations in both directions for all + new syntax, lowering it to extension nodes or attributes and then lifting it + back out. *) + +(** How did we choose between using the attribute embedding and the extension + node embedding for a particular syntactic category? + + Generally, we prefer the attribute embedding: it's more compatible with + ppxes that aren't aware of Jane Syntax. (E.g., if a type looks like a tuple, + it truly is a tuple and not an extension node embedding.) + + We can't apply the attribute embedding everywhere because some syntactic + categories, like structure items, don't carry attributes. For these, we + use extension nodes. + + However, the attribute embedding is more inconvenient in some ways than + the extension node embedding. For example, the attribute embedding requires + callers to strip out Jane Syntax-related attributes from the attribute list + before processing it. We've tried to make this obvious from the signature + of, say, [Jane_syntax.Expression.of_ast], but this is somewhat more + inconvenient than just operating on the [expr_desc]. Nonetheless, because + of the advantages with ppxlib interoperability, we've opted for the + attribute embedding where possible. +*) + +open Parsetree + +(** We carefully regulate which bindings we import from [Language_extension] to + ensure that we can import this file into places like ocamlformat or the Jane + Street internal repo with no changes. +*) +module Language_extension = struct + include Language_extension_kernel + include ( + Language_extension + : Language_extension_kernel.Language_extension_for_jane_syntax) +end + +(** For the same reason, we don't want this file to depend on new additions to + [Misc] or similar utility libraries, so we define any generic utility + functionality in this module. *) +module Util : sig + val split_last_opt : 'a list -> ('a list * 'a) option + (* Like [Misc.split_last], but doesn't throw any exceptions. *) + + val find_map_last_and_split : + f:('a -> 'b option) -> 'a list -> ('a list * 'b * 'a list) option + (* [find_map_last_and_split ~f l] returns a triple [pre, y, post] such + that [l = pre @ x @ post], [f x = Some y], and for all [x'] in + [post], [f x' = None]. If, for all [z] in [l], [f z = None], then + it returns [None]. *) +end = struct + let split_last_opt = function + | [] -> None + | (_ :: _) as xs -> Some (Misc.split_last xs) + + let find_map_last_and_split = + let rec go post ~f = function + | [] -> None + | x :: xs -> match f x with + | Some y -> Some (List.rev xs, y, post) + | None -> go (x :: post) ~f xs + in + fun ~f xs -> go [] ~f (List.rev xs) +end + +(******************************************************************************) + +module Feature : sig + type t = + | Language_extension : _ Language_extension.t -> t + | Builtin + + type error = + | Disabled_extension : _ Language_extension.t -> error + | Unknown_extension of string + + val describe_uppercase : t -> string + + val describe_lowercase : t -> string + + val extension_component : t -> string + + val of_component : string -> (t, error) result + + val is_erasable : t -> bool +end = struct + type t = Language_extension : _ Language_extension.t -> t + | Builtin + + type error = + | Disabled_extension : _ Language_extension.t -> error + | Unknown_extension of string + + let builtin_component = "_builtin" + + let describe ~uppercase = function + | Language_extension ext -> + (if uppercase then "T" else "t") ^ "he extension \"" ^ + Language_extension.to_string ext ^ "\"" + | Builtin -> + (if uppercase then "B" else "b") ^ "uilt-in syntax" + + let describe_uppercase = describe ~uppercase:true + let describe_lowercase = describe ~uppercase:false + + let extension_component = function + | Language_extension ext -> Language_extension.to_string ext + | Builtin -> builtin_component + + let of_component str = + if String.equal str builtin_component then + Ok Builtin + else + match Language_extension.of_string str with + | Some (Pack ext) -> + if Language_extension.is_enabled ext + then Ok (Language_extension ext) + else Error (Disabled_extension ext) + | None -> + Error (Unknown_extension str) + + let is_erasable = function + | Language_extension ext -> Language_extension.is_erasable ext + (* Builtin syntax changes don't involve additions or changes to concrete + syntax and are always erasable. + *) + | Builtin -> true +end + +(** Was this embedded as an [[%extension_node]] or an [[@attribute]]? Not + exported. *) +module Embedding_syntax = struct + type t = + | Extension_node + | Attribute + + let name = function + | Extension_node -> "extension node" + | Attribute -> "attribute" + + let name_indefinite = function + | Extension_node -> "an extension node" + | Attribute -> "an attribute" + + let name_plural = function + | Extension_node -> "extension nodes" + | Attribute -> "attributes" + + let pp ppf (t, name) = + let sigil = match t with + | Extension_node -> "%" + | Attribute -> "@" + in + Format.fprintf ppf "[%s%s]" sigil name +end + +(******************************************************************************) + +module Misnamed_embedding_error = struct + type t = + | No_erasability + | No_feature + | Unknown_erasability of string + + let to_string = function + | No_erasability -> "Missing erasability and feature components" + | No_feature -> "Missing a feature component" + | Unknown_erasability str -> + Printf.sprintf + "Unrecognized component where erasability was expected: `%s'" + str +end + +(** The component of an attribute or extension name that identifies whether or + not the embedded syntax is *erasable*; that is, whether or not the + upstream OCaml compiler can safely interpret the AST while ignoring the + attribute or extension. (This means that syntax encoded as extension + nodes should always be non-erasable.) Tools that consume the parse tree + we generate can make use of this information; for instance, ocamlformat + will use it to guide how we present code that can be run with both our + compiler and the upstream compiler, and ppxlib can use it to decide + whether it's ok to allow ppxes to construct syntax that uses this + emedding. In particular, the upstream version of ppxlib will allow ppxes + to produce [[@jane.erasable.*]] attributes, but will report an error if a + ppx produces a [[@jane.non_erasable.*]] attribute. + + As mentioned above, unlike for attributes, the erasable/non-erasable + distinction is not meaningful for extension nodes, as the compiler will + always error if it sees an uninterpreted extension node. So, for purposes + of tools in the wider OCaml ecosystem, it is irrelevant whether embeddings + that use extension nodes indicate [Erasable] or [Non_erasable] for this + component, but the semantically correct choice and the one we've settled + on is to use [Non_erasable]. *) +module Erasability = struct + type t = + | Erasable + | Non_erasable + + let equal t1 t2 = match t1, t2 with + | Erasable, Erasable | Non_erasable, Non_erasable -> true + | (Erasable | Non_erasable), _ -> false + + let to_string = function + | Erasable -> "erasable" + | Non_erasable -> "non_erasable" + + let of_string = function + | "erasable" -> Ok Erasable + | "non_erasable" -> Ok Non_erasable + | _ -> Error () +end + +(** An AST-style representation of the names used when generating extension + nodes or attributes for modular syntax; see the .mli file for more + details. *) +module Embedded_name : sig + + (** A nonempty list of name components, without the first two components. + (That is, without the leading root component that identifies it as part of + the modular syntax mechanism, and without the next component that + identifies the erasability.) See the .mli file for more details. *) + type components = ( :: ) of string * string list + + type t = + { erasability : Erasability.t + ; components : components + } + + (** See the mli. *) + val of_feature : Feature.t -> string list -> t + + val components : t -> components + + (** Convert one of these Jane syntax names to the embedded string form used in + the OCaml AST as the name of an extension node or an attribute; not + exposed. *) + val to_string : t -> string + + (** Parse a Jane syntax name from the OCaml AST, either as the name of an + extension node or an attribute: + - [Some (Ok _)] if it's a legal Jane-syntax name; + - [Some (Error _)] if the root is present, but the name has fewer than 3 + components or the erasability component is malformed; and + - [None] if it doesn't start with the leading root name and isn't part + of our Jane-syntax machinery. + Not exposed. *) + val of_string : string -> (t, Misnamed_embedding_error.t) result option + + val marker_attribute_handler : + string list -> (loc:Location.t -> attribute) + * (attributes -> attributes option) + * (attributes -> bool) + + (** Checks whether a name is a "marker attribute name", as created by + [marker_attribute_handler] (see the .mli file). Used to avoid trying to + desguar them as normal Jane syntax. Not exposed. *) + val is_marker : t -> bool + + (** Print out the embedded form of a Jane-syntax name, in quotes; for use in + error messages. *) + val pp_quoted_name : Format.formatter -> t -> unit + + (** Print out an empty extension node or attribute with a Jane-syntax name, + accompanied by an indefinite article; for use in error messages. Not + exposed. *) + val pp_a_term : Format.formatter -> Embedding_syntax.t * t -> unit +end = struct + (** The three parameters that control how we encode Jane-syntax extension node + names. When updating these, update comments that refer to them by their + contents! *) + module Config = struct + (** The separator between name components *) + let separator = '.' + + (** The leading namespace that identifies this extension node or attribute + as reserved for a piece of modular syntax *) + let root = "jane" + + (** For printing purposes, the appropriate indefinite article for [root] *) + let article = "a" + end + + include Config + + let separator_str = String.make 1 separator + + type components = ( :: ) of string * string list + + type t = + { erasability : Erasability.t + ; components : components + } + + let of_feature feature trailing_components = + let feature_component = Feature.extension_component feature in + let erasability : Erasability.t = + if Feature.is_erasable feature then Erasable else Non_erasable + in + { erasability; components = feature_component :: trailing_components } + + let components t = t.components + + let to_string { erasability; components = feat :: subparts } = + String.concat + separator_str + (root :: Erasability.to_string erasability :: feat :: subparts) + + let of_string str : (t, Misnamed_embedding_error.t) result option = + match String.split_on_char separator str with + | root' :: parts when String.equal root root' -> begin + match parts with + | [] -> Some (Error No_erasability) + | [_] -> Some (Error No_feature) + | erasability :: feat :: subparts -> begin + match Erasability.of_string erasability with + | Ok erasability -> + Some (Ok { erasability; components = feat :: subparts }) + | Error () -> Some (Error (Unknown_erasability erasability)) + end + end + | _ :: _ | [] -> None + + let marker_component = "_marker" + + let marker_attribute_handler components = + let t = + { erasability = Erasable; components = marker_component :: components } + in + let make ~loc = + let loc = Location.ghostify loc in + Ast_helper.Attr.mk ~loc (Location.mkloc (to_string t) loc) (PStr []) + in + let is_t = function + | { attr_name = { txt = name; loc = _ } + ; attr_payload = PStr [] + ; attr_loc = _ } -> + String.equal (to_string t) name + | _ -> false + in + let extract attrs = + attrs |> + Util.find_map_last_and_split + ~f:(fun attr -> if is_t attr then Some () else None) |> + Option.map (fun (pre, (), post) -> pre @ post) + in + let has = List.exists is_t in + make, extract, has + + let is_marker = function + | { erasability = Erasable; components = feature :: _ } -> + String.equal feature marker_component + | _ -> false + + let pp_quoted_name ppf t = Format.fprintf ppf "\"%s\"" (to_string t) + + let pp_a_term ppf (esyn, t) = + Format.fprintf ppf "%s %a" article Embedding_syntax.pp (esyn, to_string t) +end + +(******************************************************************************) +module Error = struct + (** Someone used [[%jane.*.FEATNAME]]/[[@jane.*.FEATNAME]] wrong *) + type malformed_embedding = + | Has_payload of payload + + (** An error triggered when desugaring a language extension from an OCaml + AST; should always be fatal *) + type error = + | Malformed_embedding of + Embedding_syntax.t * Embedded_name.t * malformed_embedding + | Unknown_extension of Embedding_syntax.t * Erasability.t * string + | Disabled_extension : + { ext : _ Language_extension.t + ; maturity : Language_extension.maturity option + } -> error + | Wrong_syntactic_category of Feature.t * string + | Misnamed_embedding of + Misnamed_embedding_error.t * string * Embedding_syntax.t + | Bad_introduction of Embedding_syntax.t * Embedded_name.t + | Missing_location_attribute + + (** The exception type thrown when desugaring a piece of modular syntax from + an OCaml AST *) + exception Error of Location.t * error +end + +open Error + +let assert_extension_enabled + (type a) ~loc (ext : a Language_extension.t) (setting : a) + = + if not (Language_extension.is_at_least ext setting) then + let maturity : Language_extension.maturity option = + match ext with + | Layouts -> Some (setting : Language_extension.maturity) + | _ -> None + in + raise (Error(loc, Disabled_extension { ext; maturity })) +;; + +let report_error ~loc = function + | Malformed_embedding(what, name, malformed) -> begin + match malformed with + | Has_payload _payload -> + Location.errorf + ~loc + "@[Modular syntax %s are not allowed to have a payload,@ \ + but %a does@]" + (Embedding_syntax.name_plural what) + Embedded_name.pp_quoted_name name + end + | Unknown_extension (what, erasability, name) -> + let embedded_name = { Embedded_name.erasability; components = [name] } in + Location.errorf + ~loc + "@[Unknown extension \"%s\" referenced via@ %a %s@]" + name + Embedded_name.pp_a_term (what, embedded_name) + (Embedding_syntax.name what) + | Disabled_extension { ext; maturity } -> begin + (* CR layouts: The [maturity] special case is a bit ad-hoc, but the + layouts error message would be much worse without it. It also + would be nice to mention the language construct in the error message. + *) + match maturity with + | None -> + Location.errorf + ~loc + "The extension \"%s\" is disabled and cannot be used" + (Language_extension.to_string ext) + | Some maturity -> + Location.errorf + ~loc + "This construct requires the %s version of the extension \"%s\", \ + which is disabled and cannot be used" + (Language_extension.maturity_to_string maturity) + (Language_extension.to_string ext) + end + | Wrong_syntactic_category(feat, cat) -> + Location.errorf + ~loc + "%s cannot appear in %s" + (Feature.describe_uppercase feat) + cat + | Misnamed_embedding (err, name, what) -> + Location.errorf + ~loc + "Cannot have %s named %a: %s" + (Embedding_syntax.name_indefinite what) + Embedding_syntax.pp (what, name) + (Misnamed_embedding_error.to_string err) + | Bad_introduction(what, ({ components = ext :: _; _ } as name)) -> + Location.errorf + ~loc + "@[The extension \"%s\" was referenced improperly; it started with@ \ + %a %s,@ not %a one@]" + ext + Embedded_name.pp_a_term (what, name) + (Embedding_syntax.name what) + Embedded_name.pp_a_term (what, { name with components = [ext] }) + | Missing_location_attribute -> + Location.errorf + ~loc + "@[All attribute embeddings are expected to contain a location \ + attribute,@ but one was missing here.@]" + +let () = + Location.register_error_of_exn + (function + | Error(loc, err) -> Some (report_error ~loc err) + | _ -> None) + +(******************************************************************************) +(** Generically find and create the OCaml AST syntax used to encode one of our + novel syntactic features. One module per variety of AST (expressions, + patterns, etc.). *) + +(** The parameters that define how to look for [[%jane.*.FEATNAME]] and + [[@jane.*.FEATNAME]] inside ASTs of a certain syntactic category. This + module type describes the input to the [Make_with_attribute] and + [Make_with_extension_node] functors (though they stipulate additional + requirements for their inputs). +*) +module type AST_syntactic_category = sig + (** The AST type (e.g., [Parsetree.expression]) *) + type ast + + (** The name for this syntactic category in the plural form; used for error + messages (e.g., "expressions") *) + val plural : string + + (** How to get the location attached to an AST node. Should just be + [fun tm -> tm.pCAT_loc] for the appropriate syntactic category [CAT]. *) + val location : ast -> Location.t + + (** Set the location of an AST node. *) + val with_location : ast -> Location.t -> ast +end + +module type AST_internal = sig + type 'ast with_attributes + + include AST_syntactic_category + + val embedding_syntax : Embedding_syntax.t + + val make_jane_syntax : Embedded_name.t -> ast -> ast + + (** Given an AST node, check if it's a representation of a term from one of + our novel syntactic features; if it is, split it back up into its name and + the body. If the embedded term is malformed in any way, raises an error; + if the input isn't an embedding of one of our novel syntactic features, + returns [None]. Partial inverse of [make_jane_syntax]. *) + val match_jane_syntax : ast -> (Embedded_name.t * ast with_attributes) option +end + +module type AST_with_attributes_internal = sig + include AST_internal with type 'ast with_attributes := 'ast * attributes + val add_attributes : attributes -> ast -> ast + val set_attributes : ast -> attributes -> ast +end + +(* Parses the embedded name from an embedding, raising if + the embedding is malformed. Malformed means either: + + 1. The embedding has a payload; attribute payloads must + be empty, so other ppxes can traverse "into" them. + + 2. NAME is missing; i.e., the attribute is just [[@jane]] or + [[@jane.ERASABILITY]], and similarly for extension nodes. +*) +let parse_embedding_exn ~loc ~payload ~name ~embedding_syntax = + let raise_error err = raise (Error (loc, err)) in + match Embedded_name.of_string name with + | Some (Ok name) when Embedded_name.is_marker name -> None + | Some (Ok name) -> begin + let raise_malformed err = + raise_error (Malformed_embedding (embedding_syntax, name, err)) + in + match payload with + | PStr [] -> Some name + | _ -> raise_malformed (Has_payload payload) + end + | Some (Error err) -> + raise_error (Misnamed_embedding (err, name, embedding_syntax)) + | None -> None + +(** Extracts the last attribute (in list order) that was inserted by the Jane + Syntax framework, and returns the rest of the attributes in the same + relative order as was input. The attributes that come before the extracted + one are first, and the attributes that come after are last; this last + component is guaranteed not to have any Jane Syntax attributes in it. *) +let find_and_remove_jane_syntax_attribute = + Util.find_map_last_and_split + ~f:(fun { attr_name = { txt = name; loc }; attr_payload = payload } -> + parse_embedding_exn ~loc ~payload ~name ~embedding_syntax:Attribute) + +module Desugaring_error = struct + type error = + | Wrong_embedding of Embedded_name.t + | Non_embedding + | Bad_embedding of string list + | Unexpected_attributes of attributes + + exception Error of Location.t * Feature.t * error + + let report_term_for_feature ppf feature = + Format.fprintf ppf "term for@ %s" (Feature.describe_lowercase feature) + + let report_error ~loc ~feature = function + | Wrong_embedding name -> + Location.errorf ~loc + "Tried to desugar the embedded term %a@ \ + as part of a %a, a different feature" + Embedded_name.pp_quoted_name name + report_term_for_feature feature + | Non_embedding -> + Location.errorf ~loc + "Tried to desugar a non-embedded expression as part of a %a" + report_term_for_feature feature + | Bad_embedding subparts -> + Location.errorf ~loc + "Unknown, unexpected, or malformed embedded %a at %a" + report_term_for_feature + feature + Embedded_name.pp_quoted_name + (Embedded_name.of_feature feature subparts) + | Unexpected_attributes attrs -> + Location.errorf ~loc + "Non-Jane-syntax attributes were present \ + at internal Jane-syntax points as part@ of a %a@.\ + The attributes had the following names:@ %a" + report_term_for_feature + feature + (Format.pp_print_list + ~pp_sep:(fun ppf () -> Format.fprintf ppf ",@ ") + (fun ppf attr -> Format.fprintf ppf "\"%s\"" attr.attr_name.txt)) + attrs + + let () = + Location.register_error_of_exn + (function + | Error(loc, feature, err) -> Some (report_error ~loc ~feature err) + | _ -> None) +end + +(** For a syntactic category, produce translations into and out of + our novel syntax, using parsetree attributes as the encoding. +*) +module Make_with_attribute + (AST_syntactic_category : sig + include AST_syntactic_category + + val attributes : ast -> attributes + val set_attributes : ast -> attributes -> ast + end) : AST_with_attributes_internal + with type ast = AST_syntactic_category.ast += struct + include AST_syntactic_category + + let add_attributes attrs ast = set_attributes ast (attributes ast @ attrs) + + let embedding_syntax = Embedding_syntax.Attribute + + let make_attr loc name = + let loc = Location.ghostify loc in + { attr_name = { txt = Embedded_name.to_string name; loc } + ; attr_loc = loc + ; attr_payload = PStr [] + } + + let make_jane_syntax name ast = + let attr = make_attr !Ast_helper.default_loc name in + let attrs = + match Embedded_name.components name with + | [feature_component] -> + (* Outermost; save the location *) + let save_loc = location ast in + [ make_attr + save_loc + { name + with components = + [ feature_component + ; "_location" + ; if save_loc.loc_ghost then "_ghost" else "_nonghost"] + } + ; attr ] + | _ :: _ :: _ -> + [attr] + in + add_attributes attrs ast + + let restore_location_from_attr (name : Embedded_name.t) ast = + match name with + | { erasability; components = [feature_component] } -> + let raise_error err = raise (Error(location ast, err)) in + begin match Util.split_last_opt (attributes ast) with + | Some ( attrs + , { attr_name = { txt; loc } + ; attr_loc + ; attr_payload = PStr [] } ) -> begin + match Embedded_name.of_string txt with + | Some (Ok + { erasability = loc_erasability + ; components = [ loc_feature_component + ; "_location" + ; ghostiness ] + }) + when (* Checks about the outer match, deferred so that + [Misnamed_embedding] is raised preferentially *) + loc.loc_ghost && + Location.equal loc attr_loc && + (* Checks about the inner match *) + String.equal feature_component loc_feature_component && + Erasability.equal erasability loc_erasability -> + let restored_loc = + { loc with loc_ghost = match ghostiness with + | "_ghost" -> true + | "_nonghost" -> false + | _ -> raise_error Missing_location_attribute } + in + with_location (set_attributes ast attrs) restored_loc + | Some (Error err) -> + raise_error (Misnamed_embedding (err, txt, Attribute)) + | _ -> raise_error Missing_location_attribute + end + | _ -> raise_error Missing_location_attribute + end + | { erasability = _; components = _ :: _ :: _ } -> + ast + + let match_jane_syntax ast = + match find_and_remove_jane_syntax_attribute (attributes ast) with + | None -> None + | Some (inner_attrs, name, outer_attrs) -> + Some (name, + (restore_location_from_attr name @@ + set_attributes ast inner_attrs, + outer_attrs)) +end + +(** For a syntactic category, produce translations into and out of + our novel syntax, using extension nodes as the encoding. +*) +module Make_with_extension_node + (AST_syntactic_category : sig + include AST_syntactic_category + + (** How to construct an extension node for this AST (something of the + shape [[%name]]). Should just be [Ast_helper.CAT.extension] for the + appropriate syntactic category [CAT]. (This means that [?loc] should + default to [!Ast_helper.default_loc.].) *) + val make_extension_node : + ?loc:Location.t -> ?attrs:attributes -> extension -> ast + + (** Given an extension node (as created by [make_extension_node]) with an + appropriately-formed name and a body, combine them into the special + syntactic form we use for novel syntactic features in this syntactic + category. Partial inverse of [match_extension_use]. *) + val make_extension_use : extension_node:ast -> ast -> ast + + (** Given an AST node, check if it's of the special syntactic form + indicating that this is one of our novel syntactic features (as + created by [make_extension_node]), split it back up into the extension + node and the possible body. Doesn't do any checking about the + name/format of the extension or the possible body terms (for which see + [AST.match_extension]). Partial inverse of [make_extension_use]. *) + val match_extension_use : ast -> (extension * ast) option + end) : AST_internal with type ast = AST_syntactic_category.ast + and type 'ast with_attributes := 'ast = +struct + include AST_syntactic_category + + let embedding_syntax = Embedding_syntax.Extension_node + + let make_jane_syntax name ast = + make_extension_use + ast + ~extension_node: + (make_extension_node + ({ txt = Embedded_name.to_string name + ; loc = !Ast_helper.default_loc }, + PStr [])) + + let match_jane_syntax ast = + match match_extension_use ast with + | None -> None + | Some (({txt = name; loc = ext_loc}, ext_payload), body) -> + match + parse_embedding_exn + ~loc:ext_loc + ~payload:ext_payload + ~name + ~embedding_syntax + with + | None -> None + | Some name -> Some (name, body) +end + +(** The AST parameters for every subset of types; embedded as + [[[%jane.FEATNAME] * BODY]]. *) +module Type_AST_syntactic_category = struct + type ast = core_type + + (* Missing [plural] *) + + let location typ = typ.ptyp_loc + let with_location typ l = { typ with ptyp_loc = l } + + let attributes typ = typ.ptyp_attributes + let set_attributes typ ptyp_attributes = { typ with ptyp_attributes } +end + +(** Types; embedded as [[[%jane.FEATNAME] * BODY]]. *) +module Core_type0 = Make_with_attribute (struct + include Type_AST_syntactic_category + + let plural = "types" +end) + +(** Constructor arguments; the same as types, but used in fewer places *) +module Constructor_argument0 = Make_with_attribute (struct + include Type_AST_syntactic_category + + let plural = "constructor arguments" +end) + +(** Expressions; embedded using an attribute on the expression. *) +module Expression0 = Make_with_attribute (struct + type ast = expression + + let plural = "expressions" + let location expr = expr.pexp_loc + let with_location expr l = { expr with pexp_loc = l } + + let attributes expr = expr.pexp_attributes + let set_attributes expr pexp_attributes = { expr with pexp_attributes } +end) + +(** Patterns; embedded using an attribute on the pattern. *) +module Pattern0 = Make_with_attribute (struct + type ast = pattern + + let plural = "patterns" + let location pat = pat.ppat_loc + let with_location pat l = { pat with ppat_loc = l } + + let attributes pat = pat.ppat_attributes + let set_attributes pat ppat_attributes = { pat with ppat_attributes } +end) + +(** Module types; embedded using an attribute on the module type. *) +module Module_type0 = Make_with_attribute (struct + type ast = module_type + + let plural = "module types" + let location mty = mty.pmty_loc + let with_location mty l = { mty with pmty_loc = l } + + let attributes mty = mty.pmty_attributes + let set_attributes mty pmty_attributes = { mty with pmty_attributes } +end) + +(** Extension constructors; embedded using an attribute. *) +module Extension_constructor0 = Make_with_attribute (struct + type ast = extension_constructor + + let plural = "extension constructors" + let location ext = ext.pext_loc + let with_location ext l = { ext with pext_loc = l } + + let attributes ext = ext.pext_attributes + let set_attributes ext pext_attributes = { ext with pext_attributes } +end) + +(** Signature items; embedded as + [include sig [%%extension.EXTNAME];; BODY end]. Signature items don't have + attributes or we'd use them instead. +*) +module Signature_item0 = Make_with_extension_node (struct + type ast = signature_item + + let plural = "signature items" + + let location sigi = sigi.psig_loc + let with_location sigi l = { sigi with psig_loc = l } + + let make_extension_node = Ast_helper.Sig.extension + + let make_extension_use ~extension_node sigi = + Ast_helper.Sig.include_ + { pincl_mod = Ast_helper.Mty.signature [extension_node; sigi] + ; pincl_loc = !Ast_helper.default_loc + ; pincl_attributes = [] } + + let match_extension_use sigi = + match sigi.psig_desc with + | Psig_include + { pincl_mod = + { pmty_desc = + Pmty_signature + [ { psig_desc = Psig_extension (ext, []); _ } + ; sigi ] + ; _} + ; _} + -> + Some (ext, sigi) + | _ -> None +end) + +(** Structure items; embedded as + [include struct [%%extension.EXTNAME];; BODY end]. Structure items don't + have attributes or we'd use them instead. +*) +module Structure_item0 = Make_with_extension_node (struct + type ast = structure_item + + let plural = "structure items" + + let location stri = stri.pstr_loc + let with_location stri l = { stri with pstr_loc = l } + + let make_extension_node = Ast_helper.Str.extension + + let make_extension_use ~extension_node stri = + Ast_helper.Str.include_ + { pincl_mod = Ast_helper.Mod.structure [extension_node; stri] + ; pincl_loc = !Ast_helper.default_loc + ; pincl_attributes = [] } + + let match_extension_use stri = + match stri.pstr_desc with + | Pstr_include + { pincl_mod = + { pmod_desc = + Pmod_structure + [ { pstr_desc = Pstr_extension (ext, []); _ } + ; stri ] + ; _} + ; _} + -> + Some (ext, stri) + | _ -> None +end) + +(******************************************************************************) +(* Main exports *) + +module type AST = sig + type 'a with_attributes + type ast + + val make_jane_syntax : Feature.t -> string list -> ast -> ast + val make_entire_jane_syntax : + loc:Location.t -> Feature.t -> (unit -> ast) -> ast + val match_jane_syntax_piece : + Feature.t -> (ast -> string list -> 'a option) -> ast -> 'a + val make_of_ast + : of_ast_internal:(Feature.t -> ast -> 'a option) + -> (ast -> ('a with_attributes) option) +end + +module type AST_without_attributes = + AST with type 'ast with_attributes := 'ast + +module type AST_with_attributes = sig + include AST with type 'ast with_attributes := 'ast * attributes + + val add_attributes : attributes -> ast -> ast +end + +module type Handle_attributes = sig + type 'ast t + val map_ast : f:('ast1 -> 'ast2) -> 'ast1 t -> 'ast2 t + val assert_no_attributes : + loc:Location.t -> feature:Feature.t -> 'ast t -> 'ast +end + +module Uses_attributes = struct + type 'ast t = 'ast * attributes + let map_ast ~f (ast, attrs) = (f ast, attrs) + let assert_no_attributes ~loc ~feature = function + | ast, [] -> ast + | _, (_ :: _ as attrs) -> + raise (Desugaring_error.Error (loc, feature, Unexpected_attributes attrs)) +end + +module Uses_extensions = struct + type 'ast t = 'ast + let map_ast ~f = f + let assert_no_attributes ~loc:_ ~feature:_ ast = ast +end + +module Make_ast + (Handle_attributes : Handle_attributes) + (AST : AST_internal + with type 'ast with_attributes := 'ast Handle_attributes.t) + : AST with type ast = AST.ast + and type 'ast with_attributes := 'ast Handle_attributes.t = +struct + include AST + + let make_jane_syntax feature trailing_components ast = + AST.make_jane_syntax + (Embedded_name.of_feature feature trailing_components) + ast + + let make_entire_jane_syntax ~loc feature ast = + AST.with_location + (make_jane_syntax feature [] + (Ast_helper.with_default_loc (Location.ghostify loc) ast)) + loc + + (** Generically lift our custom ASTs for our novel syntax from OCaml ASTs. *) + let make_of_ast ~of_ast_internal = + let of_ast ast = + let loc = AST.location ast in + let raise_error err = raise (Error (loc, err)) in + match AST.match_jane_syntax ast with + | Some ({ erasability; components = [name] }, ast_attrs) -> begin + match Feature.of_component name with + | Ok feat -> Some begin + ast_attrs |> Handle_attributes.map_ast ~f:(fun ast -> + match of_ast_internal feat ast with + | Some ext_ast -> ext_ast + | None -> + raise_error (Wrong_syntactic_category(feat, AST.plural))) + end + | Error err -> raise_error begin match err with + | Disabled_extension ext -> + Disabled_extension { ext; maturity = None } + | Unknown_extension name -> + Unknown_extension (AST.embedding_syntax, erasability, name) + end + end + | Some ({ components = _ :: _ :: _; _ } as name, _) -> + raise_error (Bad_introduction(AST.embedding_syntax, name)) + | None -> None + in + of_ast + + let match_jane_syntax_piece feature match_subparts ast = + let raise_error err = + raise (Desugaring_error.Error(location ast, feature, err)) + in + match AST.match_jane_syntax ast with + | Some (embedded_name, ast_attrs) -> begin + let ast' = + Handle_attributes.assert_no_attributes + ~loc:(location ast) ~feature ast_attrs + in + match Embedded_name.components embedded_name with + | extension_string :: subparts + when String.equal + extension_string + (Feature.extension_component feature) -> begin + match match_subparts ast' subparts with + | Some ext_ast -> ext_ast + | None -> raise_error (Bad_embedding subparts) + end + | _ -> raise_error (Wrong_embedding embedded_name) + end + | None -> raise_error Non_embedding +end + +module Make_extension_ast + : functor (AST : AST_internal with type 'ast with_attributes := 'ast) + -> AST_without_attributes with type ast = AST.ast = + Make_ast (Uses_extensions) + +module Make_attribute_ast (AST : AST_with_attributes_internal) + : AST_with_attributes with type ast = AST.ast = +struct + include Make_ast (Uses_attributes) (AST) + let add_attributes = AST.add_attributes +end + +module Expression = Make_attribute_ast(Expression0) +module Pattern = Make_attribute_ast(Pattern0) +module Module_type = Make_attribute_ast(Module_type0) +module Signature_item = Make_extension_ast(Signature_item0) +module Structure_item = Make_extension_ast(Structure_item0) +module Core_type = Make_attribute_ast(Core_type0) +module Extension_constructor = Make_attribute_ast(Extension_constructor0) + +module Constructor_argument = struct + include Make_attribute_ast(Constructor_argument0) + + let make_of_ast ~of_ast_internal ast = + match make_of_ast ~of_ast_internal ast with + | Some (jast, []) -> Some jast + | None -> None + | Some (_, _ :: _) -> + Misc.fatal_errorf "Constructor arguments should not have attributes" +end diff --git a/vendor/parser-standard/jane_syntax_parsing.mli b/vendor/parser-standard/jane_syntax_parsing.mli new file mode 100644 index 0000000000..22e868e992 --- /dev/null +++ b/vendor/parser-standard/jane_syntax_parsing.mli @@ -0,0 +1,310 @@ +(** This module handles the logic around the syntax of our extensions to OCaml + for Jane Street, keeping the gory details wrapped up behind a clean + interface. + + As we've started to work on syntactic extensions to OCaml, three concerns + arose about the mechanics of how we wanted to maintain these changes in our + fork. + + 1. We don't want to extend the AST for our fork, as we really want to make + sure things like ppxen are cross-compatible between upstream and our + fork. Thankfully, OCaml already provides places to add extra syntax: + extension nodes and annotations! Thus, we have to come up with a way of + representing our new syntactic constructs in terms of these constructs. + + 2. We don't want to actually match on extension nodes or attributes whose + names are specific strings all over the compiler; that's incredibly + messy, and it's easy to miss cases, etc. + + 3. We want to keep our different novel syntactic features distinct so that + we can add them to upstream independently, work on them separately, and + so on. + + We have come up with a design that addresses those concerns by providing + both a nice compiler-level interface for working with our syntactic + extensions as first-class AST nodes, as well as a uniform scheme for + translating this to and from OCaml AST values by using extension nodes or + attributes. One wrinkle is that OCaml has many ASTs, one for each syntactic + category (expressions, patterns, etc.); we have to provide this facility for + each syntactic category where we want to provide extensions. A smaller + wrinkle is that our novel syntactic features come in two varieties: + *language extensions* (e.g., comprehensions) and *built-in features* (e.g., + syntactic function arity). While the former can be disabled, the latter are + parse tree changes we rely on (though they won't therefore show up in + surface syntax). + + a. For each novel syntactic feature, we will define a module (e.g., + [Comprehensions]), in which we define a proper AST type per syntactic + category we care about (e.g., [Comprehensions.expression] and its + subcomponents). This addresses concern (3); we've now contained each + separate feature (and the built-in changes) in a module. But just doing + that would leave them too siloed, so… + + b. We define an *overall auxiliary AST* for each syntactic category that's + just for our novel syntactic features; for expressions, it's called + [Jane_syntax.Expression.t]. It contains one constructor for each of the + AST types defined as described in design point (1). This addresses + concern (2); we can now match on actual OCaml constructors, as long as we + can get ahold of them. And to do that… + + c. We define a general scheme for how we represent our novel syntactic + features in terms of the existing ASTs, and provide a few primitives for + consuming/creating AST nodes of this form, for each syntactic category. + There's not a lot of abstraction to be done, or at least it's not (yet) + apparent what abstraction there is to do, so most of this remains manual. + (Setting up a full lens-based/otherwise bidirectional approach sounds + like a great opportunity for yak-shaving, but not *actually* a good + idea.) This solves concern (3), and by doing it uniformly helps us + address multiple cases at one stroke. + + Then, for each syntactic category, we define a module (in + [jane_syntax_parsing.ml]) that contains functions for converting between the + [Parsetree] representation and the higher-level representation. These + modules are inhabitants of [AST.t], and the [AST] module exposes operations + on them. + + This module contains the logic for moving to and from OCaml ASTs; the gory + details of the encoding are detailed in the implementation. All the actual + ASTs should live in [Jane_syntax], which is the only module that should + directly depend on this one. + + When using this module, we often want to specify what our syntax extensions + look like when desugared into OCaml ASTs, so that we can validate the + translation code. We generally specify this as a BNF grammar, but we don't + want to depend on the specific details of the desugaring. Thus, instead of + writing out extension nodes or attributes directly, we write the result of + [Some_ast.make_extension ~loc [name1; name2; ...; NameN] a] as the special + syntax [{% 'name1.name2.....nameN' | a %}] in the BNF. Other pieces of the + OCaml AST are used as normal. + + One detail which we hide as much as possible is locations: whenever + constructing an OCaml AST node -- whether with [wrap_desc], the functions in + [Ast_helper], or some other way -- the location should be left to be + defaulted (and the default, [!Ast_helper.make_default], should be ghost). + The [make_entire_jane_syntax] function will handle making sure this default + location is set appropriately. If this isn't done and any locations on + subterms aren't marked as ghost, the compiler will work fine, but ppxlib may + detect that you've violated its well-formedness constraints and fail to + parse the resulting AST. *) + +(******************************************************************************) + +(** The type enumerating our novel syntactic features, which are either a + language extension (separated out by which one) or the collection of all + built-in features. *) +module Feature : sig + type t = + | Language_extension : _ Language_extension.t -> t + | Builtin + + (** The component of an attribute or extension name that identifies the + feature. This is the third component. + *) + val extension_component : t -> string +end + +(** An AST-style representation of the names used when generating extension + nodes or attributes for modular syntax. We use this to abstract over the + details of how they're encoded, so we have some flexibility in changing them + (although comments may refer to the specific encoding choices). This is + also why we don't expose any functions for rendering or parsing these names; + that's all handled internally. *) +module Embedded_name : sig + (** A nonempty list of name components, without the first two components. + (That is, without the leading root component that identifies it as part of + the modular syntax mechanism, and without the next component that + identifies the erasability.) + + This is a nonempty list corresponding to the different components of the + name: first the feature, and then any subparts. + *) + type components = ( :: ) of string * string list + + type t + + (** Creates an embedded name whose erasability component is whether the + feature is erasable, and whose feature component is the feature's name. + The second argument is treated as the trailing components after the + feature name. + *) + val of_feature : Feature.t -> string list -> t + + (** Extract the components from an embedded name; just includes the + user-specified components, not the leading or erasability components, as + with the [components] type. *) + val components : t -> components + + (** Create a new "marker attribute". These are Jane-syntax-style attributes, + but exist outside of the full Jane syntax machinery; they can be added + directly to syntax nodes, aren't matched on and turned into ASTs, and so + on and so forth. The format of the attribute name is not guaranteed to be + stable across compiler versions, but it will end with the specified + components as if they were the second part of a [components] value. + + Given [let make, extract, has = marker_attribute_handler comps], then: + + - [make ~loc] creates the specified marker attribute at the [ghost] + version of the provided location. + - [extract attrs] pulls out the specified marker attribute from [attrs], + and returns all the other attributes if it was present. If the specified + marker attribute was not present, returns [None]. + - [has attrs] returns [true] if the list of attributes contains the + specified marker attribute, and [false] otherwise. It's equivalent to + [Option.is_some (extract attrs)]. *) + val marker_attribute_handler : + string list -> (loc:Location.t -> Parsetree.attribute) + * (Parsetree.attributes -> Parsetree.attributes option) + * (Parsetree.attributes -> bool) + + (** Print out the embedded form of a Jane-syntax name, in quotes; for use in + error messages. *) + val pp_quoted_name : Format.formatter -> t -> unit +end + +(** Each syntactic category that contains novel syntactic features has a + corresponding module of this module type. We're adding these lazily as we + need them. When you add another one, make sure also to add special handling + in [Ast_iterator] and [Ast_mapper]. + + This module type comes in two varieties: [AST_with_attributes] and + [AST_without_attributes]. They reflect whether desugaring an OCaml AST into + our extended one should ([with]) or shouldn't ([without]) return the + attributes as well. This choice is recorded in the [with_attributes] + type. *) +module type AST = sig + (** The AST type (e.g., [Parsetree.expression]) *) + type ast + + (** Embed a term from one of our novel syntactic features in the AST using the + given name (in the [Feature.t]) and body (the [ast]). Any locations in + the generated AST will be set to [!Ast_helper.default_loc], which should + be [ghost]. The list of components should be nonempty; if it's empty, you + probably want [make_entire_jane_syntax] instead. *) + val make_jane_syntax + : Feature.t + -> string list + -> ast + -> ast + + (** As [make_jane_syntax], but specifically for the AST node corresponding to + the entire piece of novel syntax (e.g., for a list comprehension, the + whole [[x for x in xs]], and not a subcomponent like [for x in xs]). The + provided location is used for the location of the resulting AST node. + Additionally, [Ast_helper.default_loc] is set locally to the [ghost] + version of that location, which is why the [ast] is generated from a + function call; it is during this call that the location is so set. *) + val make_entire_jane_syntax + : loc:Location.t + -> Feature.t + -> (unit -> ast) + -> ast + + (** Given a *nested* term from one of our novel syntactic features that has + *already* been embedded in the AST by [make_jane_syntax], matches on the + name and AST of that embedding to lift it back to the Jane syntax AST. By + "nested", this means the term ought to be a subcomponent of a + [make_entire_jane_syntax]-created term, created specifically by + [make_jane_syntax] with a nonempty list of components. + + For example, to distinguish between the different terms in the + [-extension local] expression AST, we write: + + {[ + let of_expr = + Expression.match_jane_syntax_piece feature @@ fun expr -> function + | ["local"] -> Some (Lexp_local expr) + | ["exclave"] -> Some (Lexp_exclave expr) + | _ -> None + ]} + *) + val match_jane_syntax_piece + : Feature.t -> (ast -> string list -> 'a option) -> ast -> 'a + + (** How to attach attributes to the result of [make_of_ast]. Will either + return a pair (see [AST_with_attributes]) or will simply be equal to ['a] + when there are no attributes ([AST_without_attributes]). *) + type 'a with_attributes + + (** Build an [of_ast] function. The return value of this function should be + used to implement [of_ast] in modules satisfying the signature + [Jane_syntax.AST]. + + The returned function interprets an AST term in the specified syntactic + category as a term of the appropriate auxiliary extended AST if possible. + It raises an error if it finds a term from a disabled extension or if the + embedding is malformed. + *) + val make_of_ast + : of_ast_internal:(Feature.t -> ast -> 'a option) + (** A function to convert [Parsetree]'s AST to our novel extended one. The + choice of feature and the piece of syntax will both be extracted from + the embedding by the first argument. + + If the given syntax feature does not actually extend the given syntactic + category, returns [None]; this will be reported as an error. (For + example: There are no pattern comprehensions, so when building the + extended pattern AST, this function will return [None] if it spots an + embedding that claims to be from [Language_extension Comprehensions].) + *) + -> (ast -> 'a with_attributes option) +end + +(** An [AST] that keeps track of attributes. This also includes + attribute-manipulating functions. *) +module type AST_with_attributes = sig + include AST with type 'ast with_attributes := 'ast * Parsetree.attributes + + (** Add attributes to an AST term, appending them to the attributes already + present. *) + val add_attributes : Parsetree.attributes -> ast -> ast +end + +(** An [AST] that does not keep track of attributes. *) +module type AST_without_attributes = + AST with type 'ast with_attributes := 'ast + +module Expression : + AST_with_attributes with type ast = Parsetree.expression + +module Pattern : + AST_with_attributes with type ast = Parsetree.pattern + +module Module_type : + AST_with_attributes with type ast = Parsetree.module_type + +module Signature_item : + AST_without_attributes with type ast = Parsetree.signature_item + +module Structure_item : + AST_without_attributes with type ast = Parsetree.structure_item + +module Core_type : + AST_with_attributes with type ast = Parsetree.core_type + +module Constructor_argument : + AST_without_attributes with type ast = Parsetree.core_type + +module Extension_constructor : + AST_with_attributes with type ast = Parsetree.extension_constructor + +(** Require that an extension is enabled for at least the provided level, or + else throw an exception (of an abstract type) at the provided location + saying otherwise. This is intended to be used in [jane_syntax.ml] when a + certain piece of syntax requires two extensions to be enabled at once (e.g., + immutable array comprehensions such as [[:x for x = 1 to 10:]], which + require both [Comprehensions] and [Immutable_arrays]). *) +val assert_extension_enabled : + loc:Location.t -> 'a Language_extension.t -> 'a -> unit + +(** Errors around the representation of our extended ASTs. These should mostly + just be fatal, but they're needed for one test case + (language-extensions/language_extensions.ml). *) +module Error : sig + (** An error triggered when desugaring a piece of embedded novel syntax from + an OCaml AST; left abstract because it should always be fatal *) + type error + + (** The exception type thrown when desugaring a piece of extended syntax from + an OCaml AST *) + exception Error of Location.t * error +end diff --git a/vendor/parser-standard/language_extension.ml b/vendor/parser-standard/language_extension.ml new file mode 100644 index 0000000000..141e8b7f7b --- /dev/null +++ b/vendor/parser-standard/language_extension.ml @@ -0,0 +1,286 @@ +include Language_extension_kernel + +(* operations we want on every extension level *) +module type Extension_level = sig + type t + val compare : t -> t -> int + val max : t -> t -> t + val max_value : t + val all : t list + val to_command_line_suffix : t -> string +end + +module Unit = struct + type t = unit + let compare = Unit.compare + let max _ _ = () + let max_value = () + let all = [()] + let to_command_line_suffix () = "" +end + +module Maturity = struct + type t = maturity = Stable | Beta | Alpha + + let compare t1 t2 = + let rank = function + | Stable -> 1 + | Beta -> 2 + | Alpha -> 3 + in + compare (rank t1) (rank t2) + + let max t1 t2 = if compare t1 t2 >= 0 then t1 else t2 + let max_value = Alpha + + let all = [ Stable; Beta; Alpha ] + + let to_command_line_suffix = function + | Stable -> "" + | Beta -> "_beta" + | Alpha -> "_alpha" +end + +let get_level_ops : type a. a t -> (module Extension_level with type t = a) = + function + | Comprehensions -> (module Unit) + | Local -> (module Unit) + | Include_functor -> (module Unit) + | Polymorphic_parameters -> (module Unit) + | Immutable_arrays -> (module Unit) + | Module_strengthening -> (module Unit) + | Layouts -> (module Maturity) + | SIMD -> (module Unit) + +type extn_pair = Exist_pair.t = Pair : 'a t * 'a -> extn_pair +type exist = Exist.t = Pack : _ t -> exist + +(**********************************) +(* string conversions *) + +let pair_of_string_exn extn_name = match pair_of_string extn_name with + | Some pair -> pair + | None -> + raise (Arg.Bad(Printf.sprintf "Extension %s is not known" extn_name)) + +(************************************) +(* equality *) + +let equal_t (type a b) (a : a t) (b : b t) : (a, b) Misc.eq option = match a, b with + | Comprehensions, Comprehensions -> Some Refl + | Local, Local -> Some Refl + | Include_functor, Include_functor -> Some Refl + | Polymorphic_parameters, Polymorphic_parameters -> Some Refl + | Immutable_arrays, Immutable_arrays -> Some Refl + | Module_strengthening, Module_strengthening -> Some Refl + | Layouts, Layouts -> Some Refl + | SIMD, SIMD -> Some Refl + | (Comprehensions | Local | Include_functor | Polymorphic_parameters | + Immutable_arrays | Module_strengthening | Layouts | SIMD), _ -> None + +let equal a b = Option.is_some (equal_t a b) + +(*****************************) +(* extension universes *) + +module Universe : sig + val is_allowed : 'a t -> bool + val check : 'a t -> unit + val check_maximal : unit -> unit + + type t = + | No_extensions + | Only_erasable + | Any + + val set : t -> bool +end = struct + (** Which extensions can be enabled? *) + type t = + | No_extensions + | Only_erasable + | Any + + let compare t1 t2 = + let rank = function + | No_extensions -> 1 + | Only_erasable -> 2 + | Any -> 3 + in + compare (rank t1) (rank t2) + + let universe = ref Any + + let compiler_options = function + | No_extensions -> "flag -disable-all-extensions" + | Only_erasable -> "flag -only-erasable-extensions" + | Any -> "default options" + + let is_allowed ext = match !universe with + | No_extensions -> false + | Only_erasable -> is_erasable ext + | Any -> true + + (* are _all_ extensions allowed? *) + let all_allowed () = match !universe with + | Any -> true + | No_extensions | Only_erasable -> false + + (* The terminating [()] argument helps protect against ignored arguments. See + the documentation for [Base.failwithf]. *) + let fail fmt = + Format.ksprintf (fun str () -> raise (Arg.Bad str)) fmt + + let check extn = + if not (is_allowed extn) + then fail "Cannot enable extension %s: incompatible with %s" + (to_string extn) + (compiler_options !universe) + () + + let check_maximal () = + if not (all_allowed ()) + then fail "Cannot enable all extensions: incompatible with %s" + (compiler_options !universe) + () + + (* returns whether or not a change was actually made *) + let set new_universe = + let cmp = compare new_universe !universe in + if cmp > 0 + then fail "Cannot specify %s: incompatible with %s" + (compiler_options new_universe) + (compiler_options !universe) + (); + universe := new_universe; + cmp <> 0 +end + +(*****************************************) +(* enabling / disabling *) + +(* Mutable state. Invariants: + + (1) [!extensions] contains at most one copy of each extension. + + (2) Every member of [!extensions] satisfies [Universe.is_allowed]. + (For instance, [!universe = No_extensions] implies + [!extensions = []]). *) + +let default_extensions : extn_pair list = + [ Pair (Local, ()) + ; Pair (Include_functor, ()) + ; Pair (Polymorphic_parameters, ()) + ] +let extensions : extn_pair list ref = ref default_extensions + +let set_worker (type a) (extn : a t) = function + | Some value -> + Universe.check extn; + let (module Ops) = get_level_ops extn in + let rec update_extensions already_seen : extn_pair list -> extn_pair list = + function + | [] -> Pair (extn, value) :: already_seen + | ((Pair (extn', v) as e) :: es) -> + match equal_t extn extn' with + | None -> update_extensions (e :: already_seen) es + | Some Refl -> + Pair (extn, Ops.max v value) :: List.rev_append already_seen es + in + extensions := update_extensions [] !extensions + | None -> + extensions := + List.filter (fun (Pair (extn', _) : extn_pair) -> not (equal extn extn')) + !extensions + +let set extn ~enabled = + set_worker extn (if enabled then Some () else None) +let enable extn value = set_worker extn (Some value) +let disable extn = set_worker extn None + +(* It might make sense to ban [set], [enable], [disable], + [only_erasable_extensions], and [disallow_extensions] inside [f], but it's + not clear that it's worth the hassle *) +let with_set_worker extn value f = + (* This is similar to [Misc.protect_refs], but we don't have values to set + [extensions] to. *) + let current_extensions = !extensions in + Fun.protect + ~finally:(fun () -> extensions := current_extensions) + (fun () -> + set_worker extn value; + f ()) + +let with_set extn ~enabled = + with_set_worker extn (if enabled then Some () else None) +let with_enabled extn value = with_set_worker extn (Some value) +let with_disabled extn = with_set_worker extn None + +let enable_of_string_exn extn_name = match pair_of_string_exn extn_name with + | Pair (extn, setting) -> enable extn setting + +let disable_of_string_exn extn_name = match pair_of_string_exn extn_name with + | Pair (extn, _) -> disable extn + +let disable_all () = + extensions := [] + +let enable_maximal () = + Universe.check_maximal (); + let maximal_pair (Pack extn) = + let (module Ops) = get_level_ops extn in + Pair (extn, Ops.max_value) + in + extensions := List.map maximal_pair Exist.all + +let restrict_to_erasable_extensions () = + let changed = Universe.set Only_erasable in + if changed + then extensions := + List.filter (fun (Pair (extn, _)) -> Universe.is_allowed extn) !extensions + +let disallow_extensions () = + ignore (Universe.set No_extensions : bool); + disable_all () + +(********************************************) +(* checking an extension *) + +let is_at_least (type a) (extn : a t) (value : a) = + let rec check : extn_pair list -> bool = function + | [] -> false + | (Pair (e, v) :: es) -> + let (module Ops) = get_level_ops e in + match equal_t e extn with + | Some Refl -> Ops.compare v value >= 0 + | None -> check es + in + check !extensions + +let is_enabled extn = + let rec check : extn_pair list -> bool = function + | [] -> false + | (Pair (e, _) :: _) when equal e extn -> true + | (_ :: es) -> check es + in + check !extensions + + +module Exist = struct + include Exist + + let to_command_line_strings (Pack extn) = + let (module Ops) = get_level_ops extn in + List.map + (fun level -> to_string extn ^ Ops.to_command_line_suffix level) + Ops.all + + let to_string : t -> string = function + | Pack extn -> to_string extn + + let is_enabled : t -> bool = function + | Pack extn -> is_enabled extn + + let is_erasable : t -> bool = function + | Pack extn -> is_erasable extn +end diff --git a/vendor/parser-standard/language_extension.mli b/vendor/parser-standard/language_extension.mli new file mode 100644 index 0000000000..e685e34959 --- /dev/null +++ b/vendor/parser-standard/language_extension.mli @@ -0,0 +1,107 @@ +(** Language extensions provided by the Jane Street version of the OCaml + compiler. +*) + +(** A setting for extensions that track multiple maturity levels *) +type maturity = Language_extension_kernel.maturity = Stable | Beta | Alpha + +(** The type of language extensions. An ['a t] is an extension that can either + be off or be set to have any value in ['a], so a [unit t] can be either on + or off, while a [maturity t] can have different maturity settings. *) +type 'a t = 'a Language_extension_kernel.t = + | Comprehensions : unit t + | Local : unit t + | Include_functor : unit t + | Polymorphic_parameters : unit t + | Immutable_arrays : unit t + | Module_strengthening : unit t + | Layouts : maturity t + | SIMD : unit t + +(** Existentially packed language extension *) +module Exist : sig + type 'a extn = 'a t (* this is removed from the sig by the [with] below; + ocamldoc doesn't like [:=] in sigs *) + type t = Language_extension_kernel.Exist.t = + | Pack : 'a extn -> t + + val to_string : t -> string + val is_enabled : t -> bool + val is_erasable : t -> bool + + (** Returns a list of all strings, like ["layouts_beta"], that + correspond to this extension. *) + val to_command_line_strings : t -> string list + + val all : t list +end with type 'a extn := 'a t + +(** Equality on language extensions *) +val equal : 'a t -> 'b t -> bool + +(** Disable all extensions *) +val disable_all : unit -> unit + +(** Maximally enable all extensions (that is, set to [Alpha] for [maturity] + extensions. *) +val enable_maximal : unit -> unit + +(** Check if a language extension is "erasable", i.e. whether it can be + harmlessly translated to attributes and compiled with the upstream + compiler. *) +val is_erasable : 'a t -> bool + +(** Print and parse language extensions; parsing is case-insensitive *) +val to_string : 'a t -> string +val of_string : string -> Exist.t option + +val maturity_to_string : maturity -> string + +(** Enable and disable according to command-line strings; these raise + an exception if the input string is invalid. *) +val enable_of_string_exn : string -> unit +val disable_of_string_exn : string -> unit + +(** Enable and disable language extensions; these operations are idempotent *) +val set : unit t -> enabled:bool -> unit +val enable : 'a t -> 'a -> unit +val disable : 'a t -> unit + +(** Check if a language extension is currently enabled (at any maturity level) +*) +val is_enabled : 'a t -> bool + +(** Check if a language extension is enabled at least at the given level *) +val is_at_least : 'a t -> 'a -> bool + +(** Tooling support: Temporarily enable and disable language extensions; these + operations are idempotent. Calls to [set], [enable], [disable], and + [disallow_extensions] inside the body of the function argument will also + be rolled back when the function finishes, but this behavior may change; + nest multiple [with_*] functions instead. *) +val with_set : unit t -> enabled:bool -> (unit -> unit) -> unit +val with_enabled : 'a t -> 'a -> (unit -> unit) -> unit +val with_disabled : 'a t -> (unit -> unit) -> unit + +(** Permanently restrict the allowable extensions to those that are + "erasable", i.e. those that can be harmlessly translated to attributes and + compiled with the upstream compiler. Used for [-only-erasable-extensions] + to ensure that some code is guaranteed to be compatible with upstream + OCaml after rewriting to attributes. When called, disables any + currently-enabled non-erasable extensions, including any that are on by + default. Causes any future uses of [set ~enabled:true], [enable], and + their [with_] variants to raise if used with a non-erasable extension. + The [is_enabled] function will still work on any extensions, it will just + always return [false] on non-erasable ones. Will raise if called after + [disallow_extensions]; the ratchet of extension restriction only goes one + way. *) +val restrict_to_erasable_extensions : unit -> unit + +(** Permanently ban all extensions; used for [-disable-all-extensions] to + ensure that some code is 100% extension-free. When called, disables any + currently-enabled extensions, including the defaults. Causes any future + uses of [set ~enabled:true], [enable], and their [with_] variants to + raise; also causes any future uses of [only_erasable_extensions] to raise. + The [is_enabled] function will still work, it will just always return + [false].*) +val disallow_extensions : unit -> unit diff --git a/vendor/parser-standard/language_extension_kernel.ml b/vendor/parser-standard/language_extension_kernel.ml new file mode 100644 index 0000000000..eae564c13d --- /dev/null +++ b/vendor/parser-standard/language_extension_kernel.ml @@ -0,0 +1,100 @@ +type maturity = Stable | Beta | Alpha + +(* Remember to update [all] when changing this type. *) +type _ t = + | Comprehensions : unit t + | Local : unit t + | Include_functor : unit t + | Polymorphic_parameters : unit t + | Immutable_arrays : unit t + | Module_strengthening : unit t + | Layouts : maturity t + | SIMD : unit t + +type 'a language_extension_kernel = 'a t + +module Exist = struct + type t = Pack : _ language_extension_kernel -> t + + let all = + [ Pack Comprehensions + ; Pack Local + ; Pack Include_functor + ; Pack Polymorphic_parameters + ; Pack Immutable_arrays + ; Pack Module_strengthening + ; Pack Layouts + ; Pack SIMD + ] +end + +module Exist_pair = struct + type t = Pair : 'a language_extension_kernel * 'a -> t +end + +(* When you update this, update [pair_of_string] below too. *) +let to_string : type a. a t -> string = function + | Comprehensions -> "comprehensions" + | Local -> "local" + | Include_functor -> "include_functor" + | Polymorphic_parameters -> "polymorphic_parameters" + | Immutable_arrays -> "immutable_arrays" + | Module_strengthening -> "module_strengthening" + | Layouts -> "layouts" + | SIMD -> "simd" + +(* converts full extension names, like "layouts_alpha" to a pair of + an extension and its maturity. For extensions that don't take an + argument, the conversion is just [Language_extension_kernel.of_string]. +*) +let pair_of_string extn_name : Exist_pair.t option = + match String.lowercase_ascii extn_name with + | "comprehensions" -> Some (Pair (Comprehensions, ())) + | "local" -> Some (Pair (Local, ())) + | "include_functor" -> Some (Pair (Include_functor, ())) + | "polymorphic_parameters" -> Some (Pair (Polymorphic_parameters, ())) + | "immutable_arrays" -> Some (Pair (Immutable_arrays, ())) + | "module_strengthening" -> Some (Pair (Module_strengthening, ())) + | "layouts" -> Some (Pair (Layouts, Stable)) + | "layouts_alpha" -> Some (Pair (Layouts, Alpha)) + | "layouts_beta" -> Some (Pair (Layouts, Beta)) + | "simd" -> Some (Pair (SIMD, ())) + | _ -> None + +let maturity_to_string = function + | Alpha -> "alpha" + | Beta -> "beta" + | Stable -> "stable" + +let of_string extn_name : Exist.t option = + match pair_of_string extn_name with + | Some (Pair (ext, _)) -> Some (Pack ext) + | None -> None + +(* We'll do this in a more principled way later. *) +(* CR layouts: Note that layouts is only "mostly" erasable, because of annoying + interactions with the pre-layouts [@@immediate] attribute like: + + type ('a : immediate) t = 'a [@@immediate] + + But we've decided to punt on this issue in the short term. +*) +let is_erasable : type a. a t -> bool = function + | Local + | Layouts -> + true + | Comprehensions + | Include_functor + | Polymorphic_parameters + | Immutable_arrays + | Module_strengthening + | SIMD -> + false + +(* See the mli. *) +module type Language_extension_for_jane_syntax = sig + type nonrec 'a t = 'a t + + val is_enabled : _ t -> bool + val is_at_least : 'a t -> 'a -> bool +end diff --git a/vendor/parser-standard/language_extension_kernel.mli b/vendor/parser-standard/language_extension_kernel.mli new file mode 100644 index 0000000000..7ccbd98160 --- /dev/null +++ b/vendor/parser-standard/language_extension_kernel.mli @@ -0,0 +1,62 @@ +(** Language extensions provided by the Jane Street version of the OCaml + compiler. + + This is the signature of the {!Language_extension_kernel} module that is + directly imported into [ppxlib_jane]. +*) + +type maturity = Stable | Beta | Alpha + +(** The type of language extensions. An ['a t] is an extension that can either + be off or be set to have any value in ['a], so a [unit t] can be either on + or off, while a [maturity t] can have different maturity settings. *) +type _ t = + | Comprehensions : unit t + | Local : unit t + | Include_functor : unit t + | Polymorphic_parameters : unit t + | Immutable_arrays : unit t + | Module_strengthening : unit t + | Layouts : maturity t + | SIMD : unit t + +module Exist : sig + type 'a extn = 'a t + type t = Pack : _ extn -> t + + val all : t list +end with type 'a extn := 'a t + +module Exist_pair : sig + type 'a extn = 'a t + type t = Pair : 'a extn * 'a -> t +end with type 'a extn := 'a t + +(** Print and parse language extensions; parsing is case-insensitive *) +val to_string : _ t -> string +val of_string : string -> Exist.t option +val pair_of_string : string -> Exist_pair.t option +val maturity_to_string : maturity -> string + +(** Check if a language extension is "erasable", i.e. whether it can be + harmlessly translated to attributes and compiled with the upstream + compiler. *) +val is_erasable : _ t -> bool + +module type Language_extension_for_jane_syntax = sig + (** This module type defines the pieces of functionality used by + {!Jane_syntax_parsing} and {!Jane_syntax} so that we can more easily + import these modules into [ppxlib_jane], without also including all of the + [Language_extension] machinery. + + It includes the stateful operations that {!Jane_syntax_parsing} relies on. + This limits the number of bindings that [ppxlib_jane] needs to have mock + implementations for. + *) + + type nonrec 'a t = 'a t + + (** Check if a language extension is currently enabled. *) + val is_enabled : _ t -> bool + val is_at_least : 'a t -> 'a -> bool +end diff --git a/vendor/parser-standard/lexer.mll b/vendor/parser-standard/lexer.mll index dcaa9d89d1..7ba5bac5af 100644 --- a/vendor/parser-standard/lexer.mll +++ b/vendor/parser-standard/lexer.mll @@ -50,12 +50,14 @@ let keyword_table = "else", ELSE; "end", END; "exception", EXCEPTION; + "exclave_", EXCLAVE; "external", EXTERNAL; "false", FALSE; "for", FOR; "fun", FUN; "function", FUNCTION; "functor", FUNCTOR; + "global_", GLOBAL; "if", IF; "in", IN; "include", INCLUDE; @@ -63,6 +65,7 @@ let keyword_table = "initializer", INITIALIZER; "lazy", LAZY; "let", LET; + "local_", LOCAL; "match", MATCH; "method", METHOD; "module", MODULE; @@ -98,6 +101,12 @@ let keyword_table = "asr", INFIXOP4("asr") ] +let lookup_keyword name = + match Hashtbl.find keyword_table name with + | kw -> kw + | exception Not_found -> + LIDENT name + (* To buffer string literals *) let string_buffer = Buffer.create 256 @@ -117,6 +126,110 @@ let is_in_string = ref false let in_string () = !is_in_string let print_warnings = ref true +let at_beginning_of_line pos = (pos.pos_cnum = pos.pos_bol) + +(* See the comment on the [directive] lexer. *) +type directive_lexing_already_consumed = + | Hash + | Hash_and_line_num of { line_num : string } + +type deferred_token = + { token : token + ; start_pos : Lexing.position + ; end_pos : Lexing.position + } + +(* This queue will only ever have 0 or 1 elements in it. We use it + instead of an [option ref] for its convenient interface. +*) +let deferred_tokens : deferred_token Queue.t = Queue.create () + +(* Effectively splits the text in the lexer's current "window" (defined below) + into two halves. The current call to the lexer will return the first half of + the text in the window, and the next call to the lexer will return the second + half (of length [len]) of the text in the window. + + "window" refers to the text matched by a production of the lexer. It spans + from [lexer.lex_start_p] to [lexer.lex_curr_p]. + + The function accomplishes this splitting by doing two things: + - It sets the current window of the lexbuf to only account for the + first half of the text. (The first half is of length: |text|-len.) + - It enqueues a token into [deferred_tokens] such that, the next time the + lexer is called, it will return the specified [token] *and* set the window + of the lexbuf to account for the second half of the text. (The second half + is of length: |text|.) + + This business with setting the window of the lexbuf is only so that error + messages point at the right place in the program text. +*) +let enqueue_token_from_end_of_lexbuf_window (lexbuf : Lexing.lexbuf) token ~len = + let suffix_end = lexbuf.lex_curr_p in + let suffix_start = + { suffix_end with pos_cnum = suffix_end.pos_cnum - len } + in + lexbuf.lex_curr_p <- suffix_start; + Queue.add + { token; start_pos = suffix_start; end_pos = suffix_end } + deferred_tokens + +(* Note [Lexing hack for float#] + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + This note describes a non-backward-compatible Jane Street--internal change to + the lexer. + + We want the lexer to lex [float#] differently than [float #]. [float#] is the + new syntax for the unboxed float type. It veers close to the syntax for the + type of all objects belonging to a class [c], which is [#c]. The way we + navigate this veering is by producing the following tokens for these source + program examples, where LIDENT(s) is an LIDENT with text [s]. + + float#c ==> LIDENT(float) HASH_SUFFIX LIDENT(c) + float# c ==> LIDENT(float) HASH_SUFFIX LIDENT(c) + float # c ==> LIDENT(float) HASH LIDENT(c) + float #c ==> LIDENT(float) HASH LIDENT(c) + + (A) The parser interprets [LIDENT(float) HASH_SUFFIX LIDENT(c)] as + "the type constructor [c] applied to the unboxed float type." + (B) The parser interprets [LIDENT(float) HASH LIDENT(c)] as + "the type constructor [#c] applied to the usual boxed float type." + + This is not a backward-compatible change. In upstream ocaml, the lexer + produces [LIDENT(float) HASH LIDENT(c)] for all the above source programs. + + But, this isn't problematic: everybody puts a space before '#c' to mean (B). + No existing code writes things like [float#c] or indeed [float# c]. + + We accomplish this hack by setting some global mutable state upon seeing + an identifier immediately followed by a hash. When that state is set, we + will produce [HASH_SUFFIX] the next time the lexer is called. This is + done in [enqueue_hash_suffix_from_end_of_lexbuf_window]. + + Note [Lexing hack for hash operators] + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + To complicate the above story, we don't want to treat the # in the + below program as HASH_SUFFIX: + + x#~#y + + We instead want: + + x#~#y ==> LIDENT(x) HASHOP(#~#) LIDENT(y) + + This is to allow for infix hash operators. We add an additional hack, in + the style of Note [Lexing hack for float#], where the lexer consumes [x#~#] + all at once, but produces LIDENT(x) from the current call to the lexer and + HASHOP(#~#) from the next call to the lexer. This is done in + [enqueue_hashop_from_end_of_lexbuf_window]. + *) + +let enqueue_hash_suffix_from_end_of_lexbuf_window lexbuf = + enqueue_token_from_end_of_lexbuf_window lexbuf HASH_SUFFIX ~len:1 + +let enqueue_hashop_from_end_of_lexbuf_window lexbuf ~hashop = + enqueue_token_from_end_of_lexbuf_window lexbuf (HASHOP hashop) + ~len:(String.length hashop) + (* Escaped chars are interpreted in strings unless they are in comments. *) let store_escaped_char lexbuf c = if in_comment () then store_lexeme lexbuf else store_string_char c @@ -159,6 +272,25 @@ let wrap_comment_lexer comment lexbuf = let error lexbuf e = raise (Error(e, Location.curr lexbuf)) let error_loc loc e = raise (Error(e, loc)) +let directive_error + (lexbuf : Lexing.lexbuf) explanation ~directive ~already_consumed + = + let directive_prefix = + match already_consumed with + | Hash -> "#" + | Hash_and_line_num { line_num } -> "#" ^ line_num + in + (* Set the lexbuf's current window to extend to the start of + the directive so the error message's location is more accurate. + *) + lexbuf.lex_start_p <- + { lexbuf.lex_start_p with + pos_cnum = + lexbuf.lex_start_p.pos_cnum - String.length directive_prefix + }; + error lexbuf + (Invalid_directive (directive_prefix ^ directive, Some explanation)) + (* to translate escape sequences *) let digit_value c = @@ -303,6 +435,18 @@ let add_docstring_comment ds = let comments () = List.rev !comment_list +let float ~maybe_hash lit modifier = + match maybe_hash with + | "#" -> HASH_FLOAT (lit, modifier) + | "" -> FLOAT (lit, modifier) + | unexpected -> fatal_error ("expected # or empty string: " ^ unexpected) + +let int ~maybe_hash lit modifier = + match maybe_hash with + | "#" -> HASH_INT (lit, modifier) + | "" -> INT (lit, modifier) + | unexpected -> fatal_error ("expected # or empty string: " ^ unexpected) + (* Error report *) open Format @@ -435,23 +579,62 @@ rule token = parse | "?" (lowercase_latin1 identchar_latin1 * as name) ':' { warn_latin1 lexbuf; OPTLABEL name } + (* Lowercase identifiers are split into 3 cases, and the order matters + (longest to shortest). + *) + | (lowercase identchar * as name) ('#' symbolchar_or_hash+ as hashop) + (* See Note [Lexing hack for hash operators] *) + { enqueue_hashop_from_end_of_lexbuf_window lexbuf ~hashop; + lookup_keyword name } + | (lowercase identchar * as name) '#' + (* See Note [Lexing hack for float#] *) + { enqueue_hash_suffix_from_end_of_lexbuf_window lexbuf; + lookup_keyword name } | lowercase identchar * as name { try Hashtbl.find keyword_table name with Not_found -> LIDENT name } + (* Lowercase latin1 identifiers are split into 3 cases, and the order matters + (longest to shortest). + *) + | (lowercase_latin1 identchar_latin1 * as name) + ('#' symbolchar_or_hash+ as hashop) + (* See Note [Lexing hack for hash operators] *) + { warn_latin1 lexbuf; + enqueue_hashop_from_end_of_lexbuf_window lexbuf ~hashop; + LIDENT name } + | (lowercase_latin1 identchar_latin1 * as name) '#' + (* See Note [Lexing hack for float#] *) + { warn_latin1 lexbuf; + enqueue_hash_suffix_from_end_of_lexbuf_window lexbuf; + LIDENT name } | lowercase_latin1 identchar_latin1 * as name { warn_latin1 lexbuf; LIDENT name } | uppercase identchar * as name { UIDENT name } (* No capitalized keywords *) | uppercase_latin1 identchar_latin1 * as name { warn_latin1 lexbuf; UIDENT name } - | int_literal as lit { INT (lit, None) } - | (int_literal as lit) (literal_modifier as modif) - { INT (lit, Some modif) } - | float_literal | hex_float_literal as lit - { FLOAT (lit, None) } - | (float_literal | hex_float_literal as lit) (literal_modifier as modif) - { FLOAT (lit, Some modif) } - | (float_literal | hex_float_literal | int_literal) identchar+ as invalid + (* This matches either an integer literal or a directive. If the text "#2" + appears at the beginning of a line that lexes as a directive, then it + should be treated as a directive and not an unboxed int. This is acceptable + because "#2" isn't a valid unboxed int anyway because it lacks a suffix; + the parser rejects unboxed-ints-lacking-suffixes with a more descriptive + error message. + *) + | ('#'? as maybe_hash) (int_literal as lit) + { if at_beginning_of_line lexbuf.lex_start_p && maybe_hash = "#" then + try directive (Hash_and_line_num { line_num = lit }) lexbuf + with Failure _ -> int ~maybe_hash lit None + else int ~maybe_hash lit None + } + | ('#'? as maybe_hash) (int_literal as lit) (literal_modifier as modif) + { int ~maybe_hash lit (Some modif) } + | ('#'? as maybe_hash) + (float_literal | hex_float_literal as lit) + { float ~maybe_hash lit None } + | ('#'? as maybe_hash) + (float_literal | hex_float_literal as lit) (literal_modifier as modif) + { float ~maybe_hash lit (Some modif) } + | '#'? (float_literal | hex_float_literal | int_literal) identchar+ as invalid { error lexbuf (Invalid_literal invalid) } | "\"" { let s, loc = wrap_string_lexer string lexbuf in @@ -536,10 +719,9 @@ rule token = parse STAR } | "#" - { let at_beginning_of_line pos = (pos.pos_cnum = pos.pos_bol) in - if not (at_beginning_of_line lexbuf.lex_start_p) + { if not (at_beginning_of_line lexbuf.lex_start_p) then HASH - else try directive lexbuf with Failure _ -> HASH + else try directive Hash lexbuf with Failure _ -> HASH } | "&" { AMPERSAND } | "&&" { AMPERAMPER } @@ -564,6 +746,7 @@ rule token = parse | "=" { EQUAL } | "[" { LBRACKET } | "[|" { LBRACKETBAR } + | "[:" { LBRACKETCOLON } | "[<" { LBRACKETLESS } | "[>" { LBRACKETGREATER } | "]" { RBRACKET } @@ -572,6 +755,7 @@ rule token = parse | "|" { BAR } | "||" { BARBAR } | "|]" { BARRBRACKET } + | ":]" { COLONRBRACKET } | ">" { GREATER } | ">]" { GREATERRBRACKET } | "}" { RBRACE } @@ -615,14 +799,31 @@ rule token = parse | (_ as illegal_char) { error lexbuf (Illegal_character illegal_char) } -and directive = parse - | ([' ' '\t']* (['0'-'9']+ as _num) [' ' '\t']* - ("\"" ([^ '\010' '\013' '\"' ] * as _name) "\"") as directive) +(* An example of a directive is: + +#4 "filename.ml" + + Here, 4 is the line number and filename.ml is the file name. The '#' must + appear in column 0. + + The [directive] lexer is called when some portion of the start of + the line was already consumed, either just the '#' or the '#4'. That's + indicated by the [already_consumed] argument. The caller is responsible + for checking that the '#' appears in column 0. + + The [directive] lexer always attempts to read the line number from the + lexbuf. It expects to receive a line number from exactly one source (either + the lexbuf or the [already_consumed] argument, but not both) and will fail if + this isn't the case. +*) +and directive already_consumed = parse + | ([' ' '\t']* (['0'-'9']+? as _line_num_opt) [' ' '\t']* + ("\"" ([^ '\010' '\013' '\"' ] * as _name) "\"") as directive) [^ '\010' '\013'] * { (* Line directives are not preserved by the lexer so we error out. *) let explanation = "line directives are not supported" in - error lexbuf (Invalid_directive ("#" ^ directive, Some explanation)) + directive_error lexbuf explanation ~already_consumed ~directive } and comment = parse "(*" @@ -783,6 +984,13 @@ and skip_hash_bang = parse | "" { () } { + let token lexbuf = + match Queue.take_opt deferred_tokens with + | None -> token lexbuf + | Some { token; start_pos; end_pos } -> + lexbuf.lex_start_p <- start_pos; + lexbuf.lex_curr_p <- end_pos; + token let token_with_comments lexbuf = match !preprocessor with diff --git a/vendor/parser-standard/parser.mly b/vendor/parser-standard/parser.mly index a303e14725..65eb4b59b5 100644 --- a/vendor/parser-standard/parser.mly +++ b/vendor/parser-standard/parser.mly @@ -48,7 +48,7 @@ let ghost_loc (startpos, endpos) = { let mktyp ~loc ?attrs d = Typ.mk ~loc:(make_loc loc) ?attrs d let mkpat ~loc d = Pat.mk ~loc:(make_loc loc) d -let mkexp ~loc d = Exp.mk ~loc:(make_loc loc) d +let mkexp ~loc ?attrs d = Exp.mk ~loc:(make_loc loc) ?attrs d let mkmty ~loc ?attrs d = Mty.mk ~loc:(make_loc loc) ?attrs d let mksig ~loc d = Sig.mk ~loc:(make_loc loc) d let mkmod ~loc ?attrs d = Mod.mk ~loc:(make_loc loc) ?attrs d @@ -64,8 +64,6 @@ let pstr_type ((nr, ext), tys) = (Pstr_type (nr, tys), ext) let pstr_exception (te, ext) = (Pstr_exception te, ext) -let pstr_include (body, ext) = - (Pstr_include body, ext) let pstr_recmodule (ext, bindings) = (Pstr_recmodule bindings, ext) @@ -80,8 +78,6 @@ let psig_typesubst ((nr, ext), tys) = (Psig_typesubst tys, ext) let psig_exception (te, ext) = (Psig_exception te, ext) -let psig_include (body, ext) = - (Psig_include body, ext) let mkctf ~loc ?attrs ?docs d = Ctf.mk ~loc:(make_loc loc) ?attrs ?docs d @@ -150,19 +146,44 @@ let neg_string f = let mkuminus ~oploc name arg = match name, arg.pexp_desc with | "-", Pexp_constant(Pconst_integer (n,m)) -> - Pexp_constant(Pconst_integer(neg_string n,m)) + Pexp_constant(Pconst_integer(neg_string n,m)), arg.pexp_attributes | ("-" | "-."), Pexp_constant(Pconst_float (f, m)) -> - Pexp_constant(Pconst_float(neg_string f, m)) + Pexp_constant(Pconst_float(neg_string f, m)), arg.pexp_attributes | _ -> - Pexp_apply(mkoperator ~loc:oploc ("~" ^ name), [Nolabel, arg]) + Pexp_apply(mkoperator ~loc:oploc ("~" ^ name), [Nolabel, arg]), [] let mkuplus ~oploc name arg = let desc = arg.pexp_desc in match name, desc with | "+", Pexp_constant(Pconst_integer _) - | ("+" | "+."), Pexp_constant(Pconst_float _) -> desc + | ("+" | "+."), Pexp_constant(Pconst_float _) -> desc, arg.pexp_attributes | _ -> - Pexp_apply(mkoperator ~loc:oploc ("~" ^ name), [Nolabel, arg]) + Pexp_apply(mkoperator ~loc:oploc ("~" ^ name), [Nolabel, arg]), [] +module Local_syntax_category = struct + type _ t = + | Type : core_type t + | Expression : expression t + | Pattern : pattern t +end + +let local_if : type ast. ast Local_syntax_category.t -> _ -> _ -> ast -> ast = + fun cat is_local sloc x -> + if is_local then + let make : loc:_ -> attrs:_ -> ast = match cat with + | Type -> Jane_syntax.Local.type_of (Ltyp_local x) + | Expression -> Jane_syntax.Local.expr_of (Lexp_local x) + | Pattern -> Jane_syntax.Local.pat_of (Lpat_local x) + in + make ~loc:(make_loc sloc) ~attrs:[] + else + x + +let global_if global_flag sloc carg = + match global_flag with + | Global -> + Jane_syntax.Local.constr_arg_of ~loc:(make_loc sloc) (Lcarg_global carg) + | Nothing -> + carg (* TODO define an abstraction boundary between locations-as-pairs and locations-as-Location.t; it should be clear when we move from @@ -227,6 +248,51 @@ let unclosed opening_name opening_loc closing_name closing_loc = raise(Syntaxerr.Error(Syntaxerr.Unclosed(make_loc opening_loc, opening_name, make_loc closing_loc, closing_name))) +(* Normal mutable arrays and immutable arrays are parsed identically, just with + different delimiters. The parsing is done by the [array_exprs] rule, and the + [Generic_array] module provides (1) a type representing the possible results, + and (2) a function for going from that type to an AST fragment representing + an array. *) +module Generic_array = struct + (** The three possible ways to parse an array (writing [[? ... ?]] for either + [[| ... |]] or [[: ... :]]): *) + type (_, _) t = + | Literal : 'ast list -> ('ast, 'ast_desc) t + (** A plain array literal/pattern, [[? x; y; z ?]] *) + | Opened_literal : open_declaration * + Lexing.position * + Lexing.position * + expression list + -> (expression, expression_desc) t + (** An array literal with a local open, [Module.[? x; y; z ?]] (only valid in + expressions) *) + | Unclosed : (Lexing.position * Lexing.position) * + (Lexing.position * Lexing.position) + -> (_, _) t + (** Parse error: an unclosed array literal, [\[? x; y; z] with no closing + [?\]]. *) + + let to_ast (type ast ast_desc) + (open_ : string) (close : string) + (array : ast list -> ast_desc) + : (ast, ast_desc) t -> ast_desc = function + | Literal elts -> + array elts + | Opened_literal(od, startpos, endpos, elts) -> + (Pexp_open(od, mkexp ~loc:(startpos, endpos) (array elts)) : ast_desc) + | Unclosed(startpos, endpos) -> + unclosed open_ startpos close endpos + + let expression : _ -> _ -> _ -> (expression, expression_desc) t -> _ = to_ast + let pattern : _ -> _ -> _ -> (pattern, pattern_desc) t -> _ = to_ast +end + +let ppat_iarray loc elts = + (Jane_syntax.Immutable_arrays.pat_of + ~attrs:[] + ~loc:(make_loc loc) + (Iapat_immutable_array elts)).ppat_desc + let expecting loc nonterm = raise Syntaxerr.(Error(Expecting(make_loc loc, nonterm))) @@ -304,6 +370,8 @@ let bigarray_untuplify = function { pexp_desc = Pexp_tuple explist; pexp_loc = _ } -> explist | exp -> [exp] +(* Immutable array indexing is a regular operator, so it doesn't need a special + case here *) let builtin_arraylike_name loc _ ~assign paren_kind n = let opname = if assign then "set" else "get" in let opname = if !Clflags.unsafe then "unsafe_" ^ opname else opname in @@ -633,6 +701,98 @@ let mk_directive ~loc name arg = pdir_loc = make_loc loc; } +let check_layout loc id = + begin + match id with + | ("any" | "value" | "void" | "immediate64" | "immediate" | "float64") -> () + | _ -> expecting loc "layout" + end; + let loc = make_loc loc in + Attr.mk ~loc (mkloc id loc) (PStr []) + +(* Unboxed literals *) + +(* CR layouts v2.5: The [unboxed_*] functions will both be improved and lose + their explicit assert once we have real unboxed literals in Jane syntax; they + may also get re-inlined at that point *) +let unboxed_literals_extension = Language_extension.Layouts + +module Constant : sig + type t = private + | Value of constant + | Unboxed of Jane_syntax.Unboxed_constants.t + + type loc := Lexing.position * Lexing.position + + val value : Parsetree.constant -> t + val unboxed : loc:loc -> Jane_syntax.Unboxed_constants.t -> t + val to_expression : loc:loc -> t -> expression + val to_pattern : loc:loc -> t -> pattern + val assert_is_value : loc:loc -> where:string -> t -> constant +end = struct + type t = + | Value of constant + | Unboxed of Jane_syntax.Unboxed_constants.t + + let value x = Value x + + let assert_unboxed_literals ~loc = + Language_extension.( + Jane_syntax_parsing.assert_extension_enabled ~loc Layouts Alpha) + + let unboxed ~loc x = + assert_unboxed_literals ~loc:(make_loc loc); + Unboxed x + + let to_expression ~loc : t -> expression = function + | Value const_value -> + mkexp ~loc (Pexp_constant const_value) + | Unboxed const_unboxed -> + Jane_syntax.Unboxed_constants.expr_of + ~loc:(make_loc loc) ~attrs:[] const_unboxed + + let to_pattern ~loc : t -> pattern = function + | Value const_value -> + mkpat ~loc (Ppat_constant const_value) + | Unboxed const_unboxed -> + Jane_syntax.Unboxed_constants.pat_of + ~loc:(make_loc loc) ~attrs:[] const_unboxed + + let assert_is_value ~loc ~where : t -> Parsetree.constant = function + | Value x -> x + | Unboxed _ -> + not_expecting loc (Printf.sprintf "unboxed literal %s" where) +end + +type sign = Positive | Negative + +let with_sign sign num = + match sign with + | Positive -> num + | Negative -> "-" ^ num + +let unboxed_int sloc int_loc sign (n, m) = + match m with + | Some m -> + Constant.unboxed ~loc:int_loc (Integer (with_sign sign n, m)) + | None -> + if Language_extension.is_enabled unboxed_literals_extension then + expecting int_loc "unboxed integer literal with type-specifying suffix" + else + not_expecting sloc "line number directive" + +let unboxed_float sloc sign (f, m) = + Constant.unboxed ~loc:sloc (Float (with_sign sign f, m)) + +(* Unboxed float type *) + +let assert_unboxed_float_type ~loc = + Language_extension.( + Jane_syntax_parsing.assert_extension_enabled ~loc Layouts Alpha) + +let unboxed_float_type sloc tys = + assert_unboxed_float_type ~loc:(make_loc sloc); + Ptyp_constr (mkloc (Lident "float#") (make_loc sloc), tys) %} /* Tokens */ @@ -663,6 +823,7 @@ let mk_directive ~loc name arg = %token COLONCOLON "::" %token COLONEQUAL ":=" %token COLONGREATER ":>" +%token COLONRBRACKET ":]" %token COMMA "," %token CONSTRAINT "constraint" %token DO "do" @@ -675,13 +836,16 @@ let mk_directive ~loc name arg = %token EOF "" %token EQUAL "=" %token EXCEPTION "exception" +%token EXCLAVE "exclave_" %token EXTERNAL "external" %token FALSE "false" -%token FLOAT "42.0" (* just an example *) +%token FLOAT "42.0" (* just an example *) +%token HASH_FLOAT "#42.0" (* just an example *) %token FOR "for" %token FUN "fun" %token FUNCTION "function" %token FUNCTOR "functor" +%token GLOBAL "global_" %token GREATER ">" %token GREATERRBRACE ">}" %token GREATERRBRACKET ">]" @@ -698,13 +862,15 @@ let mk_directive ~loc name arg = %token ANDOP "and*" (* just an example *) %token INHERIT "inherit" %token INITIALIZER "initializer" -%token INT "42" (* just an example *) +%token INT "42" (* just an example *) +%token HASH_INT "#42l" (* just an example *) %token LABEL "~label:" (* just an example *) %token LAZY "lazy" %token LBRACE "{" %token LBRACELESS "{<" %token LBRACKET "[" %token LBRACKETBAR "[|" +%token LBRACKETCOLON "[:" %token LBRACKETLESS "[<" %token LBRACKETGREATER "[>" %token LBRACKETPERCENT "[%" @@ -713,6 +879,7 @@ let mk_directive ~loc name arg = %token LESSMINUS "<-" %token LET "let" %token LIDENT "lident" (* just an example *) +%token LOCAL "local_" %token LPAREN "(" %token LBRACKETAT "[@" %token LBRACKETATAT "[@@" @@ -747,6 +914,7 @@ let mk_directive ~loc name arg = %token SEMI ";" %token SEMISEMI ";;" %token HASH "#" +%token HASH_SUFFIX "# " %token HASHOP "##" (* just an example *) %token SIG "sig" %token SLASH "/" @@ -804,7 +972,7 @@ The precedences must be listed from low to high. %nonassoc IN %nonassoc below_SEMI %nonassoc SEMI /* below EQUAL ({lbl=...; lbl=...}) */ -%nonassoc LET /* above SEMI ( ...; let ... in ...) */ +%nonassoc LET FOR /* above SEMI ( ...; let ... in ...) */ %nonassoc below_WITH %nonassoc FUNCTION WITH /* below BAR (match ... with ...) */ %nonassoc AND /* above WITH (module rec A: SIG with ... and ...) */ @@ -816,6 +984,8 @@ The precedences must be listed from low to high. %left BAR /* pattern (p|p|p) */ %nonassoc below_COMMA %left COMMA /* expr/expr_comma_list (e,e,e) */ +%nonassoc below_FUNCTOR /* include M */ +%nonassoc FUNCTOR /* include functor M */ %right MINUSGREATER /* function_type (t -> t -> t) */ %right OR BARBAR /* expr (e || e || e) */ %right AMPERSAND AMPERAMPER /* expr (e && e && e) */ @@ -832,13 +1002,13 @@ The precedences must be listed from low to high. %nonassoc prec_constant_constructor /* cf. simple_expr (C versus C x) */ %nonassoc prec_constr_appl /* above AS BAR COLONCOLON COMMA */ %nonassoc below_HASH -%nonassoc HASH /* simple_expr/toplevel_directive */ +%nonassoc HASH HASH_SUFFIX /* simple_expr/toplevel_directive */ %left HASHOP %nonassoc below_DOT %nonassoc DOT DOTOP /* Finally, the first tokens of simple_expr are above everything else. */ -%nonassoc BACKQUOTE BANG BEGIN CHAR FALSE FLOAT INT OBJECT - LBRACE LBRACELESS LBRACKET LBRACKETBAR LIDENT LPAREN +%nonassoc BACKQUOTE BANG BEGIN CHAR FALSE FLOAT HASH_FLOAT INT HASH_INT OBJECT + LBRACE LBRACELESS LBRACKET LBRACKETBAR LBRACKETCOLON LIDENT LPAREN NEW PREFIXOP STRING TRUE UIDENT UNDERSCORE LBRACKETPERCENT QUOTED_STRING_EXPR @@ -1453,10 +1623,18 @@ structure_item: { let (ext, l) = $1 in (Pstr_class l, ext) } | class_type_declarations { let (ext, l) = $1 in (Pstr_class_type l, ext) } - | include_statement(module_expr) - { pstr_include $1 } ) { $1 } + | include_statement(module_expr) + { let is_functor, incl, ext = $1 in + let item = + if is_functor + then Jane_syntax.Include_functor.str_item_of ~loc:(make_loc $sloc) + (Ifstr_include_functor incl) + else mkstr ~loc:$sloc (Pstr_include incl) + in + wrap_str_ext ~loc:$sloc item ext + } ; (* A single module binding. *) @@ -1532,10 +1710,17 @@ module_binding_body: (* Shared material between structures and signatures. *) +include_maybe_functor: + | INCLUDE %prec below_FUNCTOR + { false } + | INCLUDE FUNCTOR + { true } +; + (* An [include] statement can appear in a structure or in a signature, which is why this definition is parameterized. *) %inline include_statement(thing): - INCLUDE + is_functor = include_maybe_functor ext = ext attrs1 = attributes thing = thing @@ -1544,7 +1729,8 @@ module_binding_body: let attrs = attrs1 @ attrs2 in let loc = make_loc $sloc in let docs = symbol_docs $sloc in - Incl.mk thing ~attrs ~loc ~docs, ext + let incl = Incl.mk thing ~attrs ~loc ~docs in + is_functor, incl, ext } ; @@ -1647,6 +1833,9 @@ module_type: { Pmty_extension $1 } ) { $1 } + | module_type WITH mkrhs(mod_ext_longident) + { Jane_syntax.Strengthen.mty_of ~loc:(make_loc $sloc) ~attrs:[] + { mty = $1; mod_id = $3 } } ; (* A signature, which appears between SIG and END (among other places), is a list of signature elements. *) @@ -1701,14 +1890,24 @@ signature_item: { let (body, ext) = $1 in (Psig_modtypesubst body, ext) } | open_description { let (body, ext) = $1 in (Psig_open body, ext) } - | include_statement(module_type) - { psig_include $1 } | class_descriptions { let (ext, l) = $1 in (Psig_class l, ext) } | class_type_declarations { let (ext, l) = $1 in (Psig_class_type l, ext) } ) { $1 } + | include_statement(module_type) + { let is_functor, incl, ext = $1 in + let item = + if is_functor + then Jane_syntax.Include_functor.sig_item_of ~loc:(make_loc $sloc) + (Ifsig_include_functor incl) + else mksig ~loc:$sloc (Psig_include incl) + in + wrap_sig_ext ~loc:$sloc item ext + } + +; (* A module declaration. *) %inline module_declaration: @@ -2216,22 +2415,39 @@ seq_expr: mkexp ~loc:$sloc (Pexp_extension ($4, payload)) } ; labeled_simple_pattern: - QUESTION LPAREN label_let_pattern opt_default RPAREN - { (Optional (fst $3), $4, snd $3) } + QUESTION LPAREN optional_local label_let_pattern opt_default RPAREN + { (Optional (fst $4), $5, local_if Pattern $3 $loc($3) (snd $4)) } | QUESTION label_var { (Optional (fst $2), None, snd $2) } - | OPTLABEL LPAREN let_pattern opt_default RPAREN - { (Optional $1, $4, $3) } + | OPTLABEL LPAREN optional_local let_pattern opt_default RPAREN + { (Optional $1, $5, local_if Pattern $3 $loc($3) $4) } | OPTLABEL pattern_var { (Optional $1, None, $2) } - | TILDE LPAREN label_let_pattern RPAREN - { (Labelled (fst $3), None, snd $3) } + | TILDE LPAREN optional_local label_let_pattern RPAREN + { (Labelled (fst $4), None, + local_if Pattern $3 $loc($3) (snd $4)) } | TILDE label_var { (Labelled (fst $2), None, snd $2) } | LABEL simple_pattern { (Labelled $1, None, $2) } + | LABEL LPAREN LOCAL pattern RPAREN + { (Labelled $1, None, + Jane_syntax.Local.pat_of ~loc:(make_loc $loc($3)) ~attrs:[] + (Lpat_local $4) ) } | simple_pattern { (Nolabel, None, $1) } + | LPAREN LOCAL let_pattern RPAREN + { (Nolabel, None, + Jane_syntax.Local.pat_of ~loc:(make_loc $loc($2)) ~attrs:[] + (Lpat_local $3)) } + | LABEL LPAREN poly_pattern RPAREN + { (Labelled $1, None, $3) } + | LABEL LPAREN LOCAL poly_pattern RPAREN + { (Labelled $1, None, + Jane_syntax.Local.pat_of ~loc:(make_loc $loc($3)) ~attrs:[] + (Lpat_local $4)) } + | LPAREN poly_pattern RPAREN + { (Nolabel, None, $2) } ; pattern_var: @@ -2252,6 +2468,11 @@ label_let_pattern: { let lab, pat = x in lab, mkpat ~loc:$sloc (Ppat_constraint (pat, cty)) } + | x = label_var COLON + cty = mktyp (vars = typevar_list DOT ty = core_type { Ptyp_poly(vars, ty) }) + { let lab, pat = x in + lab, + mkpat ~loc:$sloc (Ppat_constraint (pat, cty)) } ; %inline label_var: mkrhs(LIDENT) @@ -2263,6 +2484,17 @@ let_pattern: | mkpat(pattern COLON core_type { Ppat_constraint($1, $3) }) { $1 } + | poly_pattern + { $1 } +; +%inline poly_pattern: + mkpat( + pat = pattern + COLON + cty = mktyp(vars = typevar_list DOT ty = core_type + { Ptyp_poly(vars, ty) }) + { Ppat_constraint(pat, cty) }) + { $1 } ; %inline indexop_expr(dot, index, right): @@ -2319,6 +2551,12 @@ expr: { not_expecting $loc($1) "wildcard \"_\"" } *) /* END AVOID */ + | LOCAL seq_expr + { Jane_syntax.Local.expr_of ~loc:(make_loc $sloc) ~attrs:[] + (Lexp_local $2) } + | EXCLAVE seq_expr + { Jane_syntax.Local.expr_of ~loc:(make_loc $sloc) ~attrs:[] + (Lexp_exclave $2) } ; %inline expr_attrs: | LET MODULE ext_attributes mkrhs(module_name) module_binding_body IN seq_expr @@ -2332,10 +2570,12 @@ expr: | FUNCTION ext_attributes match_cases { Pexp_function $3, $2 } | FUN ext_attributes labeled_simple_pattern fun_def - { let (l,o,p) = $3 in - Pexp_fun(l, o, p, $4), $2 } + { let ext, attrs = $2 in + let (l,o,p) = $3 in + Pexp_fun(l, o, p, $4), (ext, attrs) } | FUN ext_attributes LPAREN TYPE lident_list RPAREN fun_def - { (mk_newtypes ~loc:$sloc $5 $7).pexp_desc, $2 } + { let ext, attrs = $2 in + (mk_newtypes ~loc:$sloc $5 $7).pexp_desc, (ext, attrs) } | MATCH ext_attributes seq_expr WITH match_cases { Pexp_match($3, $5), $2 } | TRY ext_attributes seq_expr WITH match_cases @@ -2355,6 +2595,12 @@ expr: { Pexp_assert $3, $2 } | LAZY ext_attributes simple_expr %prec below_HASH { Pexp_lazy $3, $2 } + | subtractive expr %prec prec_unary_minus + { let desc, attrs = mkuminus ~oploc:$loc($1) $1 $2 in + desc, (None, attrs) } + | additive expr %prec prec_unary_plus + { let desc, attrs = mkuplus ~oploc:$loc($1) $1 $2 in + desc, (None, attrs) } ; %inline do_done_expr: | DO e = seq_expr DONE @@ -2373,10 +2619,6 @@ expr: { Pexp_variant($1, Some $2) } | e1 = expr op = op(infix_operator) e2 = expr { mkinfix e1 op e2 } - | subtractive expr %prec prec_unary_minus - { mkuminus ~oploc:$loc($1) $1 $2 } - | additive expr %prec prec_unary_plus - { mkuplus ~oploc:$loc($1) $1 $2 } ; simple_expr: @@ -2388,6 +2630,8 @@ simple_expr: { mkexp_constraint ~loc:$sloc $2 $3 } | indexop_expr(DOT, seq_expr, { None }) { mk_indexop_expr builtin_indexing_operators ~loc:$sloc $1 } + (* Immutable array indexing is a regular operator, so it doesn't need its own + rule and is handled by the next case *) | indexop_expr(qualified_dotop, expr_semi_list, { None }) { mk_indexop_expr user_indexing_operators ~loc:$sloc $1 } | indexop_error (DOT, seq_expr) { $1 } @@ -2397,6 +2641,7 @@ simple_expr: mkexp_attrs ~loc:$sloc desc attrs } | mkexp(simple_expr_) { $1 } + | constant { Constant.to_expression ~loc:$sloc $1 } ; %inline simple_expr_attrs: | BEGIN ext = ext attrs = attributes e = seq_expr END @@ -2418,11 +2663,94 @@ simple_expr: | OBJECT ext_attributes class_structure error { unclosed "object" $loc($1) "end" $loc($4) } ; + +comprehension_iterator: + | EQUAL expr direction_flag expr + { Jane_syntax.Comprehensions.Range { start = $2 ; stop = $4 ; direction = $3 } } + | IN expr + { Jane_syntax.Comprehensions.In $2 } +; + +comprehension_clause_binding: + | attributes pattern comprehension_iterator + { Jane_syntax.Comprehensions.{ pattern = $2 ; iterator = $3 ; attributes = $1 } } + (* We can't write [[e for local_ x = 1 to 10]], because the [local_] has to + move to the RHS and there's nowhere for it to move to; besides, you never + want that [int] to be [local_]. But we can parse [[e for local_ x in xs]]. + We have to have that as a separate rule here because it moves the [local_] + over to the RHS of the binding, so we need everything to be visible. *) + | attributes LOCAL pattern IN expr + { Jane_syntax.Comprehensions. + { pattern = $3 + ; iterator = In (Jane_syntax.Local.expr_of + ~loc:(make_loc $sloc) ~attrs:[] + (Lexp_local $5)) + ; attributes = $1 + } + } +; + +comprehension_clause: + | FOR separated_nonempty_llist(AND, comprehension_clause_binding) + { Jane_syntax.Comprehensions.For $2 } + | WHEN expr + { Jane_syntax.Comprehensions.When $2 } + +%inline comprehension(lbracket, rbracket): + lbracket expr nonempty_llist(comprehension_clause) rbracket + { Jane_syntax.Comprehensions.{ body = $2; clauses = $3 } } +; + +%inline comprehension_ext_expr: + | comprehension(LBRACKET,RBRACKET) + { Jane_syntax.Comprehensions.Cexp_list_comprehension $1 } + | comprehension(LBRACKETBAR,BARRBRACKET) + { Jane_syntax.Comprehensions.Cexp_array_comprehension (Mutable, $1) } + | comprehension(LBRACKETCOLON,COLONRBRACKET) + { Jane_syntax.Comprehensions.Cexp_array_comprehension (Immutable, $1) } +; + +%inline comprehension_expr: + comprehension_ext_expr + { (Jane_syntax.Comprehensions.expr_of + ~attrs:[] ~loc:(make_loc $sloc) $1).pexp_desc } +; + +%inline array_simple(ARR_OPEN, ARR_CLOSE, contents_semi_list): + | ARR_OPEN contents_semi_list ARR_CLOSE + { Generic_array.Literal $2 } + | ARR_OPEN contents_semi_list error + { Generic_array.Unclosed($loc($1),$loc($3)) } + | ARR_OPEN ARR_CLOSE + { Generic_array.Literal [] } +; + +%inline array_exprs(ARR_OPEN, ARR_CLOSE): + | array_simple(ARR_OPEN, ARR_CLOSE, expr_semi_list) + { $1 } + | od=open_dot_declaration DOT ARR_OPEN expr_semi_list ARR_CLOSE + { Generic_array.Opened_literal(od, $startpos($3), $endpos, $4) } + | od=open_dot_declaration DOT ARR_OPEN ARR_CLOSE + { (* TODO: review the location of Pexp_array *) + Generic_array.Opened_literal(od, $startpos($3), $endpos, []) } + | mod_longident DOT + ARR_OPEN expr_semi_list error + { Generic_array.Unclosed($loc($3), $loc($5)) } +; + +%inline array_patterns(ARR_OPEN, ARR_CLOSE): + | array_simple(ARR_OPEN, ARR_CLOSE, pattern_semi_list) + { $1 } +; + +%inline hash: + | HASH { () } + | HASH_SUFFIX { () } +; + %inline simple_expr_: | mkrhs(val_longident) { Pexp_ident ($1) } - | constant - { Pexp_constant $1 } | mkrhs(constr_longident) %prec prec_constant_constructor { Pexp_construct($1, None) } | name_tag %prec prec_constant_constructor @@ -2446,7 +2774,7 @@ simple_expr: Pexp_open(od, mkexp ~loc:$sloc (Pexp_override $4)) } | mod_longident DOT LBRACELESS object_expr_content error { unclosed "{<" $loc($3) ">}" $loc($5) } - | simple_expr HASH mkrhs(label) + | simple_expr hash mkrhs(label) { Pexp_send($1, $3) } | simple_expr op(HASHOP) simple_expr { mkinfix $1 $2 $3 } @@ -2469,24 +2797,27 @@ simple_expr: (Pexp_record(fields, exten))) } | mod_longident DOT LBRACE record_expr_content error { unclosed "{" $loc($3) "}" $loc($5) } - | LBRACKETBAR expr_semi_list BARRBRACKET - { Pexp_array($2) } - | LBRACKETBAR expr_semi_list error - { unclosed "[|" $loc($1) "|]" $loc($3) } - | LBRACKETBAR BARRBRACKET - { Pexp_array [] } - | od=open_dot_declaration DOT LBRACKETBAR expr_semi_list BARRBRACKET - { Pexp_open(od, mkexp ~loc:($startpos($3), $endpos) (Pexp_array($4))) } - | od=open_dot_declaration DOT LBRACKETBAR BARRBRACKET - { (* TODO: review the location of Pexp_array *) - Pexp_open(od, mkexp ~loc:($startpos($3), $endpos) (Pexp_array [])) } - | mod_longident DOT - LBRACKETBAR expr_semi_list error - { unclosed "[|" $loc($3) "|]" $loc($5) } + | array_exprs(LBRACKETBAR, BARRBRACKET) + { Generic_array.expression + "[|" "|]" + (fun elts -> Pexp_array elts) + $1 } + | array_exprs(LBRACKETCOLON, COLONRBRACKET) + { Generic_array.expression + "[:" ":]" + (fun elts -> + (Jane_syntax.Immutable_arrays.expr_of + ~attrs:[] + ~loc:(make_loc $sloc) + (Iaexp_immutable_array elts)).pexp_desc) + $1 } | LBRACKET expr_semi_list RBRACKET { fst (mktailexp $loc($3) $2) } | LBRACKET expr_semi_list error { unclosed "[" $loc($1) "]" $loc($3) } + | comprehension_expr { $1 } + | od=open_dot_declaration DOT comprehension_expr + { Pexp_open(od, mkexp ~loc:($loc($3)) $3) } | od=open_dot_declaration DOT LBRACKET expr_semi_list RBRACKET { let list_exp = (* TODO: review the location of list_exp *) @@ -2535,21 +2866,25 @@ labeled_simple_expr: let_binding_body_no_punning: let_ident strict_binding { ($1, $2, None) } - | let_ident type_constraint EQUAL seq_expr - { let v = $1 in (* PR#7344 *) + | optional_local let_ident type_constraint EQUAL seq_expr + { let v = $2 in (* PR#7344 *) let t = - match $2 with + match $3 with Some t, None -> Pvc_constraint { locally_abstract_univars = []; typ=t } | ground, Some coercion -> Pvc_coercion { ground; coercion} | _ -> assert false in - (v, $4, Some t) + let pat = local_if Pattern $1 $loc($1) v in + let exp = local_if Expression $1 $sloc $5 in + (pat, exp, Some t) } - | let_ident COLON poly(core_type) EQUAL seq_expr + | optional_local let_ident COLON poly(core_type) EQUAL seq_expr { - let t = ghtyp ~loc:($loc($3)) $3 in - ($1, $5, Some (Pvc_constraint { locally_abstract_univars = []; typ=t })) + let t = ghtyp ~loc:($loc($4)) $4 in + let pat = local_if Pattern $1 $loc($1) $2 in + let exp = local_if Expression $1 $sloc $6 in + (pat, exp, Some (Pvc_constraint { locally_abstract_univars = []; typ=t })) } | let_ident COLON TYPE lident_list DOT core_type EQUAL seq_expr { let constraint' = @@ -2560,6 +2895,9 @@ let_binding_body_no_punning: { ($1, $3, None) } | simple_pattern_not_ident COLON core_type EQUAL seq_expr { ($1, $5, Some(Pvc_constraint { locally_abstract_univars=[]; typ=$3 })) } + | LOCAL let_ident local_strict_binding + { ($2, Jane_syntax.Local.expr_of ~loc:(make_loc $sloc) ~attrs:[] + (Lexp_local $3), None) } ; let_binding_body: | let_binding_body_no_punning @@ -2638,6 +2976,24 @@ strict_binding: | LPAREN TYPE lident_list RPAREN fun_binding { mk_newtypes ~loc:$sloc $3 $5 } ; +local_fun_binding: + local_strict_binding + { $1 } + | type_constraint EQUAL seq_expr + { mkexp_constraint + ~loc:$sloc + (Jane_syntax.Local.expr_of ~loc:(make_loc $sloc) ~attrs:[] + (Lexp_constrain_local $3)) + $1 } +; +local_strict_binding: + EQUAL seq_expr + { $2 } + | labeled_simple_pattern local_fun_binding + { let (l, o, p) = $1 in ghexp ~loc:$sloc (Pexp_fun(l, o, p, $2)) } + | LPAREN TYPE lident_list RPAREN local_fun_binding + { mk_newtypes ~loc:$sloc $3 $5 } +; %inline match_cases: xs = preceded_or_separated_nonempty_llist(BAR, match_case) { xs } @@ -2808,19 +3164,21 @@ simple_pattern_not_ident: $3 } | mkpat(simple_pattern_not_ident_) { $1 } + | signed_constant { Constant.to_pattern $1 ~loc:$sloc } ; %inline simple_pattern_not_ident_: | UNDERSCORE { Ppat_any } - | signed_constant - { Ppat_constant $1 } | signed_constant DOTDOT signed_constant - { Ppat_interval ($1, $3) } + { let where = "in a pattern interval" in + Ppat_interval + (Constant.assert_is_value $1 ~loc:$loc($1) ~where, + Constant.assert_is_value $3 ~loc:$loc($3) ~where) } | mkrhs(constr_longident) { Ppat_construct($1, None) } | name_tag { Ppat_variant($1, None) } - | HASH mkrhs(type_longident) + | hash mkrhs(type_longident) { Ppat_type ($2) } | mkrhs(mod_longident) DOT simple_delimited_pattern { Ppat_open($1, $3) } @@ -2860,12 +3218,16 @@ simple_delimited_pattern: { fst (mktailpat $loc($3) $2) } | LBRACKET pattern_semi_list error { unclosed "[" $loc($1) "]" $loc($3) } - | LBRACKETBAR pattern_semi_list BARRBRACKET - { Ppat_array $2 } - | LBRACKETBAR BARRBRACKET - { Ppat_array [] } - | LBRACKETBAR pattern_semi_list error - { unclosed "[|" $loc($1) "|]" $loc($3) } + | array_patterns(LBRACKETBAR, BARRBRACKET) + { Generic_array.pattern + "[|" "|]" + (fun elts -> Ppat_array elts) + $1 } + | array_patterns(LBRACKETCOLON, COLONRBRACKET) + { Generic_array.pattern + "[:" ":]" + (ppat_iarray $sloc) + $1 } ) { $1 } pattern_comma_list(self): @@ -3056,12 +3418,27 @@ type_parameters: { [] } | p = type_parameter { [p] } - | LPAREN ps = separated_nonempty_llist(COMMA, type_parameter) RPAREN + | LPAREN + ps = separated_nonempty_llist(COMMA, parenthesized_type_parameter) + RPAREN { ps } ; + +layout: + ident { check_layout $loc($1) $1 } +; + +parenthesized_type_parameter: + type_parameter { $1 } + | type_variance type_variable COLON layout + { {$2 with ptyp_attributes = [$4]}, $1 } +; + type_parameter: - type_variance type_variable { $2, $1 } + type_variance type_variable attributes + { {$2 with ptyp_attributes = $3}, $1 } ; + type_variable: mktyp( QUOTE tyvar = ident @@ -3174,8 +3551,14 @@ generalized_constructor_arguments: { ($2,Pcstr_tuple [],Some $4) } ; +%inline atomic_type_gbl: + gbl = global_flag cty = atomic_type { + global_if gbl $loc(gbl) cty +} +; + constructor_arguments: - | tys = inline_separated_nonempty_llist(STAR, atomic_type) + | tys = inline_separated_nonempty_llist(STAR, atomic_type_gbl) %prec below_HASH { Pcstr_tuple tys } | LBRACE label_declarations RBRACE @@ -3187,18 +3570,33 @@ label_declarations: | label_declaration_semi label_declarations { $1 :: $2 } ; label_declaration: - mutable_flag mkrhs(label) COLON poly_type_no_attr attributes + mutable_or_global_flag mkrhs(label) COLON poly_type_no_attr attributes { let info = symbol_info $endpos in - Type.field $2 $4 ~mut:$1 ~attrs:$5 ~loc:(make_loc $sloc) ~info } + let mut, gbl = $1 in + Type.field + $2 + (global_if gbl $loc($1) $4) + ~mut + ~attrs:$5 + ~loc:(make_loc $sloc) + ~info } ; label_declaration_semi: - mutable_flag mkrhs(label) COLON poly_type_no_attr attributes SEMI attributes + mutable_or_global_flag mkrhs(label) COLON poly_type_no_attr attributes + SEMI attributes { let info = match rhs_info $endpos($5) with | Some _ as info_before_semi -> info_before_semi | None -> symbol_info $endpos in - Type.field $2 $4 ~mut:$1 ~attrs:($5 @ $7) ~loc:(make_loc $sloc) ~info } + let mut, gbl = $1 in + Type.field + $2 + (global_if gbl $loc($1) $4) + ~mut + ~attrs:($5 @ $7) + ~loc:(make_loc $sloc) + ~info } ; /* Type Extensions */ @@ -3363,14 +3761,44 @@ function_type: | ty = tuple_type %prec MINUSGREATER { ty } + | ty = strict_function_type + { ty } +; + +strict_function_type: + | mktyp( + label = arg_label + local = optional_local + domain = extra_rhs(param_type) + MINUSGREATER + codomain = strict_function_type + { Ptyp_arrow(label, local_if Type local $loc(local) domain, codomain) } + ) + { $1 } | mktyp( label = arg_label - domain = extra_rhs(tuple_type) + arg_local = optional_local + domain = extra_rhs(param_type) MINUSGREATER - codomain = function_type - { Ptyp_arrow(label, domain, codomain) } + ret_local = optional_local + codomain = tuple_type + %prec MINUSGREATER + { Ptyp_arrow(label, + local_if Type arg_local $loc(arg_local) domain, + local_if Type ret_local $loc(ret_local) + (Jane_syntax.Builtin.mark_curried + ~loc:(make_loc $loc(codomain)) codomain)) } + ) + { $1 } +; +%inline param_type: + | mktyp( + LPAREN vars = typevar_list DOT ty = core_type RPAREN + { Ptyp_poly(vars, ty) } ) { $1 } + | ty = tuple_type + { ty } ; %inline arg_label: | label = optlabel @@ -3380,6 +3808,12 @@ function_type: | /* empty */ { Nolabel } ; +%inline optional_local: + | /* empty */ + { false } + | LOCAL + { true } +; (* Tuple types include: - atomic types (see below); - proper tuple types: int * int * int list @@ -3416,7 +3850,18 @@ atomic_type: { Ptyp_any } | tys = actual_type_parameters tid = mkrhs(type_longident) - { Ptyp_constr(tid, tys) } + HASH_SUFFIX + { match tid.txt with + | Lident "float" -> + let ident_start = fst $loc(tid) in + let hash_end = snd $loc($3) in + unboxed_float_type (ident_start, hash_end) tys + | _ -> + not_expecting $sloc "Unboxed type other than float#" + } + | tys = actual_type_parameters + tid = mkrhs(type_longident) + { Ptyp_constr(tid, tys) } %prec below_HASH | LESS meth_list GREATER { let (f, c) = $2 in Ptyp_object (f, c) } | LESS GREATER @@ -3546,17 +3991,30 @@ meth_list: /* Constants */ constant: - | INT { let (n, m) = $1 in Pconst_integer (n, m) } - | CHAR { Pconst_char $1 } - | STRING { let (s, strloc, d) = $1 in Pconst_string (s, strloc, d) } - | FLOAT { let (f, m) = $1 in Pconst_float (f, m) } + | INT { let (n, m) = $1 in + Constant.value (Pconst_integer (n, m)) } + | CHAR { Constant.value (Pconst_char $1) } + | STRING { let (s, strloc, d) = $1 in + Constant.value (Pconst_string (s, strloc, d)) } + | FLOAT { let (f, m) = $1 in + Constant.value (Pconst_float (f, m)) } + | HASH_INT { unboxed_int $sloc $sloc Positive $1 } + | HASH_FLOAT { unboxed_float $sloc Positive $1 } ; signed_constant: - constant { $1 } - | MINUS INT { let (n, m) = $2 in Pconst_integer("-" ^ n, m) } - | MINUS FLOAT { let (f, m) = $2 in Pconst_float("-" ^ f, m) } - | PLUS INT { let (n, m) = $2 in Pconst_integer (n, m) } - | PLUS FLOAT { let (f, m) = $2 in Pconst_float(f, m) } + constant { $1 } + | MINUS INT { let (n, m) = $2 in + Constant.value (Pconst_integer("-" ^ n, m)) } + | MINUS FLOAT { let (f, m) = $2 in + Constant.value (Pconst_float("-" ^ f, m)) } + | MINUS HASH_INT { unboxed_int $sloc $loc($2) Negative $2 } + | MINUS HASH_FLOAT { unboxed_float $sloc Negative $2 } + | PLUS INT { let (n, m) = $2 in + Constant.value (Pconst_integer (n, m)) } + | PLUS FLOAT { let (f, m) = $2 in + Constant.value (Pconst_float(f, m)) } + | PLUS HASH_INT { unboxed_int $sloc $loc($2) Positive $2 } + | PLUS HASH_FLOAT { unboxed_float $sloc Positive $2 } ; /* Identifiers and long identifiers */ @@ -3692,7 +4150,7 @@ any_longident: /* Toplevel directives */ toplevel_directive: - HASH dir = mkrhs(ident) + hash dir = mkrhs(ident) arg = ioption(mk_directive_arg(toplevel_directive_argument)) { mk_directive ~loc:$sloc dir arg } ; @@ -3752,6 +4210,15 @@ mutable_flag: /* empty */ { Immutable } | MUTABLE { Mutable } ; +mutable_or_global_flag: + /* empty */ { Immutable, Nothing } + | MUTABLE { Mutable, Nothing } + | GLOBAL { Immutable, Global } +; +%inline global_flag: + { Nothing } + | GLOBAL { Global } +; virtual_flag: /* empty */ { Concrete } | VIRTUAL { Virtual } @@ -3838,6 +4305,7 @@ single_attr_id: | INITIALIZER { "initializer" } | LAZY { "lazy" } | LET { "let" } + | LOCAL { "local_" } | MATCH { "match" } | METHOD { "method" } | MODULE { "module" }