Commit Graph

17752 Commits (main)

Author SHA1 Message Date
Protobuf Team Bot 1f8208031a Test if a tag with unknown wire type (0x6 or 0x7) causes parsing failures.
PiperOrigin-RevId: 634878920
2024-05-17 14:03:07 -07:00
Sandy Zhang 11c27dfa4b Lazily resolve features for proto2 and proto3 for compatibility with old open source gencode that does not invoke feature resolution from gencode static init.
PiperOrigin-RevId: 634804242
2024-05-17 09:44:20 -07:00
Protobuf Team Bot 5dfdd85d34 Auto-generate files after cl/634787159 2024-05-17 15:56:35 +00:00
Protobuf Team Bot 5b8e90f404 Refactor the way we turn on the optimization in StrongPointer.
Some versions of gcc seem to advertise __cpp_nontype_template_args but not
support the argument in some cases.
Only attempt the template parameter if we are using the optimized .reloc
approach.

Fixes https://github.com/protocolbuffers/protobuf/issues/16868

PiperOrigin-RevId: 634787159
2024-05-17 08:44:35 -07:00
Protobuf Team Bot 2f3242c576 Fix ClangTidy warnings
- Call std::string::insert to insert a character.
- Use assignment instead of CopyFrom.

PiperOrigin-RevId: 634610391
2024-05-16 20:06:16 -07:00
Protobuf Team Bot a521e6e152 Internal chanegs
PiperOrigin-RevId: 634600978
2024-05-16 19:12:25 -07:00
Anton Grbin 34c843d671 [7392] [cpp] Remove dead code path for map key of type enum in JSON parsing (#16567)
# Changes

Remove dead code path -- we don't allow enums to be map keys ([proto2 spec](https://protobuf.dev/programming-guides/proto2/#maps), [proto3 spec](https://protobuf.dev/programming-guides/proto3/#maps)). In other words the case block `case FieldDescriptor::TYPE_ENUM` is dead code. Potential enum type keys will be caught in `default: return lex.Invalid("unsupported map key type");` block below similar to other unsupported map key types like double.

# Motivation

While working on fixing `IgnoreUnknownEnumStringValueInMap` conformance tests for cpp ([related issue](https://github.com/protocolbuffers/protobuf/issues/7392)) I stumbled upon a bug where we pass the wrong `field` parameter to the enum parsing function.

In this scope:
* the variable `field` is a map field of the message that holds the map. This field is not of enum type, it's a repeated message of map entires.
* the variable `key_field` is the key field of the map message entry. This field is the enum type that we need to parse here.

The function is long, so I clarified it here:

```cpp
template <typename Traits>
absl::Status ParseMap(JsonLexer& lex, Field<Traits> field, Msg<Traits>& msg) {
  (..)
  return lex.VisitObject(
      [&](LocationWith<MaybeOwnedString>& key) -> absl::Status {
          (..)
          return Traits::NewMsg(
            field, msg,
            [&](const Desc<Traits>& type, Msg<Traits>& entry) -> absl::Status {
              auto key_field = Traits::KeyField(type);
              switch (Traits::FieldType(key_field)) {
                (..)
                case FieldDescriptor::TYPE_ENUM: {
                  MaybeOwnedString key_str = key.value;
                  auto e = ParseEnumFromStr<Traits>(lex, key_str, /** bug here **/ field);
```

The correct reference should be `key_field`.

Instead of fixing the bug and leaving the dead code, it's better to remove the dead block alltogether.

Closes #16567

COPYBARA_INTEGRATE_REVIEW=https://github.com/protocolbuffers/protobuf/pull/16567 from noom:anton--7392--fix-map-key-nit d992b8a2a6
PiperOrigin-RevId: 634516984
2024-05-16 13:50:40 -07:00
Mike Kruskal 24b91a7fec Prohibit using features in the same file they're defined in.
This is an edge case we can't handle properly today.  Rather than allowing poorly defined behavior, we'll make this an error condition until we can actually support it.  In the future, it may be necessary to upgrade feature files to newer editions.

Closes #16756

PiperOrigin-RevId: 634512378
2024-05-16 13:37:35 -07:00
Sandy Zhang baeab50df0 Automated rollback of commit ec126a5069.
PiperOrigin-RevId: 634496540
2024-05-16 12:43:55 -07:00
Marcel Hlopko eca477a0f1 Internal change.
PiperOrigin-RevId: 634495582
2024-05-16 12:39:36 -07:00
Jie Luo 6c91de9de9 Nextgen Proto Pythonic API: “Add-on” proto for JSON serialize/parse
- add google.protobuf.proto_json module
- wrap json_format's MessageToDict/ParseDict to the new module:

def serialize(
    message: Message,
    always_print_fields_with_no_presence: bool=False,
    preserving_proto_field_name: bool=False,
    use_integers_for_enums: bool=False,
    descriptor_pool: Optional[DescriptorPool]=None,
    float_precision: int=None,
) -> dict

def parse(
    message_class: Type[Message],
    js_dict: dict,
    ignore_unknown_fields: bool=False,
    descriptor_pool: Optional[DescriptorPool]=None,
    max_recursion_depth: int=100
) -> Message

PiperOrigin-RevId: 634490919
2024-05-16 12:25:01 -07:00
Marcel Hlopko 69a26b217d Use the name of the rust_proto_library as the crate name
PiperOrigin-RevId: 634414403
2024-05-16 09:00:50 -07:00
Protobuf Team Bot c6cbaaeeee Devirtualize CheckTypeAndMergeFrom and use the existing function from
ClassData.
This removes a virtual function and reduces binary size.

PiperOrigin-RevId: 634413330
2024-05-16 08:57:31 -07:00
Hong Shin 0645439efa Introduce upb_EncodeStatus_String and upb_DecodeStatus_String
Wrapper languages can now use these helpers to get human-friendly
error codes as opposed to manually re-mapping.

PiperOrigin-RevId: 634112262
2024-05-15 16:30:37 -07:00
Marcel Hlopko e71c811a3a Extract crate name mangling function
PiperOrigin-RevId: 634055637
2024-05-15 13:29:52 -07:00
Protobuf Team Bot 24751b405a Add legacy Java TextFormat APIs
PiperOrigin-RevId: 634047851
2024-05-15 13:06:14 -07:00
Protobuf Team Bot ca995da263 Reduce the cost of CheckTypeAndMergeFrom in LITE_RUNTIME objects when icf is
enabled.
DownCastToGenerated injects the strong reference to T, which makes all instances of CheckTypeAndMergeFrom unique and not candidates for identical code folding.
The strong reference is not needed here in the member function.

PiperOrigin-RevId: 634033477
2024-05-15 12:08:55 -07:00
Joshua Haberman 6a2e4be010 Added C# unit tests verifying current UTF-8 behavior.
PiperOrigin-RevId: 634029739
2024-05-15 11:58:42 -07:00
Marcel Hlopko a8cb5833aa Verify rust_proto_library name
PiperOrigin-RevId: 634006714
2024-05-15 10:52:51 -07:00
Hong Shin 3574af2397 Internal Change
PiperOrigin-RevId: 634000296
2024-05-15 10:33:39 -07:00
Protobuf Team Bot cfea7596c1 Remove dead VTableName function.
PiperOrigin-RevId: 633999639
2024-05-15 10:30:55 -07:00
Protobuf Team Bot 5fdef5bfce Remove obsolete msgmut::or_default()
PiperOrigin-RevId: 633997832
2024-05-15 10:25:54 -07:00
Protobuf Team Bot 05781615a1 Internal changes/optimizations
PiperOrigin-RevId: 633982805
2024-05-15 09:39:15 -07:00
Derek Benson 9f40411caf fix unneeded mut in enum gencode
PiperOrigin-RevId: 633955047
2024-05-15 08:03:37 -07:00
Mike Kruskal 6b9a81a042 Add extension declarations for known features.
This will prevent users from accidentally overriding these with different types (e.g. https://github.com/protocolbuffers/protobuf/issues/16757 and https://github.com/protocolbuffers/protobuf/issues/16756).

PiperOrigin-RevId: 633760581
2024-05-14 17:51:20 -07:00
Sandy Zhang df630fa6f1 MODULE.bazel fixes for protobuf BCR release.
- Specifies bazel version for presubmits
- Set C++14 version (default is c++11 otherwise, which is unsupported)
- Update MODULE.bazel version + updater since publish-to-bcr can't handle constants and adds a duplicate version number

PiperOrigin-RevId: 633729225
2024-05-14 15:43:38 -07:00
Protobuf Team Bot 9b463ac048 Make the underlying type of the enum by 8-bits instead of using bitfields for
it.
It silences a warning in gcc 8/9.

PiperOrigin-RevId: 633719795
2024-05-14 15:12:09 -07:00
Protobuf Team Bot dde03553c9 Internal change
PiperOrigin-RevId: 633679764
2024-05-14 13:03:23 -07:00
Protobuf Team Bot bb68eb22dd Implement new Debug API with redaction.
Implement emittingSingleLine TextFormat printer option.

PiperOrigin-RevId: 633672722
2024-05-14 12:40:20 -07:00
Jie Luo 5a9170701e internal change
PiperOrigin-RevId: 633669104
2024-05-14 12:25:32 -07:00
Mark Hansen c6e2778f91 Replace SmallSortedMap.EmptySet with equivalent Collections.emptySet()
This reduces our code weight by a little (3 classes).

Collections.emptySet also has a singleton empty iterator, so it doesn't allocate.

PiperOrigin-RevId: 633667264
2024-05-14 12:19:26 -07:00
Protobuf Team Bot 396d661767 Add support for setting extensions with Ptr<Extension> in Upb C++ protos.
PiperOrigin-RevId: 633646216
2024-05-14 11:14:58 -07:00
Protobuf Team Bot 448e326200 Use bool HasHasbits(const FieldDescriptor*) instead of manual checks.
It is not sufficient to check schema_.HasHasbits() followed by naively skipping
repeated and oneof fields as that can miss proto3 messages with message fields
and other scalar fields without "optional" keyword.

Use internal::cpp::HasHasbits(const FieldDescriptor*) instead.

PiperOrigin-RevId: 633633184
2024-05-14 10:40:02 -07:00
pbatg 30a40eefe5 Fixed broken link in upb README.md (#16717)
I noticed this link was broken.  I'm not familiar with the project, but the code in https://github.com/protocolbuffers/protobuf/tree/main/upb/python does appear to import upb headers, so I think it's the right URL.

Closes #16717

COPYBARA_INTEGRATE_REVIEW=https://github.com/protocolbuffers/protobuf/pull/16717 from pbatg:patch-1 c9e654308d
PiperOrigin-RevId: 633589882
2024-05-14 08:34:33 -07:00
Protobuf Team Bot 2238cdc091 Add impl Clone for Rust proto owned messages.
PiperOrigin-RevId: 633265290
2024-05-13 10:57:10 -07:00
Protobuf Team Bot 7f23b700fe Introduce new API DownCastToMessage/DynamicCastToMessage as a replacement for down_cast/dynamic_cast from MessageLite to Message.
The new functions work even when RTTI is not present.

Also, add an assertion in DownCast to make sure we don't use it for message types. Message types have dedicated cast functions that should be used instead.

PiperOrigin-RevId: 633236522
2024-05-13 09:30:44 -07:00
Protobuf Team Bot fad4736785 Auto-generate files after cl/633224191 2024-05-13 16:00:07 +00:00
Sandy Zhang ce026abf92 Work around windows path length limitations by moving immutable -> full in open source (new generator paths) and shortening long file names.
Re-enable windows bazel test.

PiperOrigin-RevId: 633224191
2024-05-13 08:48:40 -07:00
Derek Benson 904266db57 inline repeated field methods
PiperOrigin-RevId: 633199701
2024-05-13 07:19:09 -07:00
Derek Benson f2d8c2bdd8 Add a reserve method on ProxiedInRepeated
PiperOrigin-RevId: 633179853
2024-05-13 05:50:14 -07:00
Mark Hansen f7a866a9ea Make IntArrayList consistent with other primitive array lists, using code generator
PiperOrigin-RevId: 633042614
2024-05-12 17:10:45 -07:00
Protobuf Team Bot ddc70b0ebf Auto-generate files after cl/632663127 2024-05-11 02:21:24 +00:00
MiguelBarro d2aa5fb483 Prevent find_package failure if multiple versions are installed (#16654)
The generated `protobuf-config-version.cmake` sets the `Protobuf_WITH_ZLIB`, `Protobuf_MSVC_STATIC_RUNTIME` and `Protobuf_BUILD_SHARED_LIBS` unconditionally while executing the `_check_and_save_build_option(...)` macro.

If there are several **Protobuf** installations available the original code may lead to failure. An example with two installations:
+ A system deployed installation with version `3.12.4`.
+ A repo's thirdparty with version `3.19.4`
The user wants it's project to use the thirdparty one by doing:
```cmake
        find_package(Protobuf 3.19.4 CONFIG REQUIRED PATHS path/to/thirdparty)
```
The user is not setting the `Protobuf_XXX` variables in its `CMakeLitsts.txt`.
If the first config processed is the system deployed it sets the variables to its desired variables, for example, `Protobuf_WITH_ZLIB=OFF` but is discarded due to the non-matched version.
Then the thirdparty's config is processed but is discarded because requires `Protobuf_WITH_ZLIB=ON` which is not a real requirement and makes the find operation fail (despite of matching version).

Closes #16654

COPYBARA_INTEGRATE_REVIEW=https://github.com/protocolbuffers/protobuf/pull/16654 from MiguelBarro:fix/cmake-config 119dd25ecb
PiperOrigin-RevId: 632664341
2024-05-10 19:17:41 -07:00
Eric Salo f93565c20c upb: publish and start using the new non-universal setters out in the wild
PiperOrigin-RevId: 632663127
2024-05-10 19:09:28 -07:00
Mike Kruskal be99767a69 Clear repository cache between bazel runs
PiperOrigin-RevId: 632621361
2024-05-10 15:42:33 -07:00
Jie Luo e7fdab4a2c Add Nextgen python proto_test to github
PiperOrigin-RevId: 632595087
2024-05-10 14:00:10 -07:00
Protobuf Team Bot 39986c099b Auto-generate files after cl/632535021 2024-05-10 17:38:46 +00:00
Protobuf Team Bot 24d42eacec Add a BUILD file under rust/accessors and rename helpers.h to default_value.h
PiperOrigin-RevId: 632535021
2024-05-10 10:26:58 -07:00
Sandy Zhang c99cf4bbfa Add `java` to reserved names to escape extensions to `java_`. This may break existing references to gencode for extensions named `java`.
This prevents shadowing of `java.lang` package commonly used in protobuf gencode. Existing extensions named `java` may or may not previously fail to compile depending on if the contents of their .proto result in gencode using `java.lang`. This is needed to fix `java_features.proto` lite gencode since enum gencode uses `java.lang`. Fields named `java` should already be escaped.

*Warning: This may break user code for existing protos with extensions named `java`. References to the extension should be renamed to use `java_` e.g. registry.add(GeneratedClassName.java_)*

PiperOrigin-RevId: 632508249
2024-05-10 08:49:22 -07:00
Jaap Taal 91b7cf3325 Fix indention formatter for google.protobuf.Any (#16785)
`google.protobuf.Any` formatting with indentation was somewhat off.

Formatting an `Any` as root object:
```json
{"@type": "type.googleapis.com/protobuf_unittest3.ForeignMessage",
"c": 1
}
```
changes to
```json
{
  "@type": "type.googleapis.com/protobuf_unittest3.ForeignMessage",
  "c": 1
}
```

For messages were `Any` is in a nested field the change makes more of a visual impact.
The `c` field seems to be at the same level as the `anyField`, but it's nested so it should be indented:
```json
{
  "anyField": {"@type": "type.googleapis.com/protobuf_unittest3.ForeignMessage",
  "c": 1
  }
}
```
changes to:
```json
{
  "anyField": {
    "@type": "type.googleapis.com/protobuf_unittest3.ForeignMessage",
    "c": 1
  }
}
```

Closes #16785

COPYBARA_INTEGRATE_REVIEW=https://github.com/protocolbuffers/protobuf/pull/16785 from q42jaap:main 72deed6ed7
PiperOrigin-RevId: 632506226
2024-05-10 08:46:04 -07:00