Skip to content

[MINOR][PYTHON][DOCS] Fixed some links in documentation. #57359

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
art000109 wants to merge 8 commits into
base: master
Choose a base branch
from
+34 −27
Open

[MINOR][PYTHON][DOCS] Fixed some links in documentation. #57359

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
0b848e4
[MINOR][PYSPARK][DOCS] Fixed some links in documentation.
Jul 19, 2026
6848b33
add links, See also, Returns sections
Jul 19, 2026
5a48090
removed DataStreamReader, DataStreamWriter from init.py
Jul 19, 2026
60aab12
Removed block 20264-20267, made links to spark.apache.org as hyperlink
Jul 20, 2026
34e77bb
Merge branch 'apache:master' into master
art000109 Jul 20, 2026
2a70036
Revert python\pyspark\sql\functions\builtin.py
Jul 20, 2026
34c8c51
Revert python\pyspark\sql\functions\builtin.py
Jul 20, 2026
579e2c0
Added a hyperlink to the Hints guide in DataFrame.hint()
Jul 20, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions python/docs/source/reference/pyspark.sql/core_classes.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,9 @@ Core Classes
DataFrameReader
DataFrameWriter
DataFrameWriterV2
MergeIntoWriter
DataStreamReader
DataStreamWriter
UDFRegistration
UDTFRegistration
udf.UserDefinedFunction
Expand Down
24 changes: 13 additions & 11 deletions python/pyspark/pandas/frame.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -390,6 +390,19 @@ class DataFrame(Frame, Generic[T]):
pandas-on-Spark DataFrame that corresponds to pandas DataFrame logically. This holds Spark
DataFrame internally.

.. versionadded:: 3.2.0

@HyukjinKwon HyukjinKwon Jul 19, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Was this added 3.2.0? Also, I think versionchanged directive as for the new parameter

@art000109 art000109 Jul 19, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, Pandas API was added in SPARK-34849 for 3.2.0

@art000109 art000109 Jul 19, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This API calls the DataFrame.pandas_api function, which was added in version 3.2.0 https://github.com/apache/spark/blob/master/python/pyspark/sql/dataframe.py#L6463

@HyukjinKwon HyukjinKwon Jul 20, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah it was pandas'


.. versionchanged:: 3.4.0
Since 3.4.0, it deals with `data` and `index` in this approach:
1, when `data` is a distributed dataset (Internal DataFrame/Spark DataFrame/
pandas-on-Spark DataFrame/pandas-on-Spark Series), it will first parallelize
the `index` if necessary, and then try to combine the `data` and `index`;
Note that if `data` and `index` doesn't have the same anchor, then
`compute.ops_on_diff_frames` should be turned on;
2, when `data` is a local dataset (Pandas DataFrame/numpy ndarray/list/etc),
it will first collect the `index` to driver if necessary, and then apply
the `pandas.DataFrame(...)` creation internally;

:ivar _internal: an internal immutable Frame to manage metadata.
:type _internal: InternalFrame

Expand All @@ -409,17 +422,6 @@ class DataFrame(Frame, Generic[T]):
copy : boolean, default False
Copy data from inputs. Only affects DataFrame / 2d ndarray input

.. versionchanged:: 3.4.0
Since 3.4.0, it deals with `data` and `index` in this approach:
1, when `data` is a distributed dataset (Internal DataFrame/Spark DataFrame/
pandas-on-Spark DataFrame/pandas-on-Spark Series), it will first parallelize
the `index` if necessary, and then try to combine the `data` and `index`;
Note that if `data` and `index` doesn't have the same anchor, then
`compute.ops_on_diff_frames` should be turned on;
2, when `data` is a local dataset (Pandas DataFrame/numpy ndarray/list/etc),
it will first collect the `index` to driver if necessary, and then apply
the `pandas.DataFrame(...)` creation internally;

Examples
--------
Constructing DataFrame from a dictionary.
Expand Down
4 changes: 3 additions & 1 deletion python/pyspark/sql/dataframe.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1206,6 +1206,8 @@ def hint(
:class:`DataFrame`
Hinted DataFrame

.. note:: See also `Hints <https://spark.apache.org/docs/latest/sql-ref-syntax-qry-select-hints.html>`

Examples
--------
>>> df = spark.createDataFrame([(2, "Alice"), (5, "Bob")], schema=["age", "name"])
Expand Down Expand Up @@ -6482,7 +6484,7 @@ def pandas_api(

See Also
--------
pyspark.pandas.frame.DataFrame.to_spark
pyspark.pandas.DataFrame.to_spark

@HyukjinKwon HyukjinKwon Jul 19, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This yes we would need thos


Examples
--------
Expand Down
2 changes: 1 addition & 1 deletion python/pyspark/sql/pandas/functions.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -74,7 +74,7 @@ def arrow_udf(f=None, returnType=None, functionType=None):
the return type of the user-defined function. The value can be either a
:class:`pyspark.sql.types.DataType` object or a DDL-formatted type string.
functionType : int, optional
an enum value in :class:`pyspark.sql.functions.ArrowUDFType`.
an enum value in :class:`~pyspark.sql.functions.ArrowUDFType`.
Default: SCALAR. This parameter exists for compatibility.
Using Python type hints is encouraged.
Expand Down
28 changes: 14 additions & 14 deletions python/pyspark/sql/tvf.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -59,7 +59,7 @@ def range(

Returns
-------
:class:`DataFrame`
:class:`~pyspark.sql.DataFrame`

Examples
--------
Expand Down Expand Up @@ -101,7 +101,7 @@ def explode(self, collection: Column) -> DataFrame:

Returns
-------
:class:`DataFrame`
:class:`~pyspark.sql.DataFrame`

See Also
--------
Expand Down Expand Up @@ -185,7 +185,7 @@ def explode_outer(self, collection: Column) -> DataFrame:

Returns
-------
:class:`DataFrame`
:class:`~pyspark.sql.DataFrame`

See Also
--------
Expand Down Expand Up @@ -238,7 +238,7 @@ def inline(self, input: Column) -> DataFrame:

Returns
-------
:class:`DataFrame`
:class:`~pyspark.sql.DataFrame`

See Also
--------
Expand Down Expand Up @@ -301,7 +301,7 @@ def inline_outer(self, input: Column) -> DataFrame:

Returns
-------
:class:`DataFrame`
:class:`~pyspark.sql.DataFrame`

See Also
--------
Expand Down Expand Up @@ -344,7 +344,7 @@ def json_tuple(self, input: Column, *fields: Column) -> DataFrame:

Returns
-------
:class:`DataFrame`
:class:`~pyspark.sql.DataFrame`

See Also
--------
Expand Down Expand Up @@ -394,7 +394,7 @@ def posexplode(self, collection: Column) -> DataFrame:

Returns
-------
:class:`DataFrame`
:class:`~pyspark.sql.DataFrame`

See Also
--------
Expand Down Expand Up @@ -437,7 +437,7 @@ def posexplode_outer(self, collection: Column) -> DataFrame:

Returns
-------
:class:`DataFrame`
:class:`~pyspark.sql.DataFrame`

See Also
--------
Expand Down Expand Up @@ -490,7 +490,7 @@ def stack(self, n: Column, *fields: Column) -> DataFrame:

Returns
-------
:class:`DataFrame`
:class:`~pyspark.sql.DataFrame`

See Also
--------
Expand Down Expand Up @@ -525,7 +525,7 @@ def collations(self) -> DataFrame:

Returns
-------
:class:`DataFrame`
:class:`~pyspark.sql.DataFrame`

Examples
--------
Expand All @@ -546,7 +546,7 @@ def sql_keywords(self) -> DataFrame:

Returns
-------
:class:`DataFrame`
:class:`~pyspark.sql.DataFrame`

Examples
--------
Expand Down Expand Up @@ -574,7 +574,7 @@ def variant_explode(self, input: Column) -> DataFrame:

Returns
-------
:class:`DataFrame`
:class:`~pyspark.sql.DataFrame`

Examples
--------
Expand Down Expand Up @@ -639,7 +639,7 @@ def variant_explode_outer(self, input: Column) -> DataFrame:

Returns
-------
:class:`DataFrame`
:class:`~pyspark.sql.DataFrame`

Examples
--------
Expand Down Expand Up @@ -681,7 +681,7 @@ def python_worker_logs(self) -> DataFrame:

Returns
-------
:class:`DataFrame`
:class:`~pyspark.sql.DataFrame`

Examples
--------
Expand Down