How to COPY library files between stages of a multi-stage Docker build while preserving symlinks?
I have a Dockerfile which is split into a two-stage multi-stage docker build. The first stage generates a basic gcc build environment in which a number of C and C++ library are compiled. The second stage uses the COPY --from=
command to copy the library files from the first stages /usr/local/lib/libproto*
to the current image's.
The problem I am seeing is that the first image contains symlinks from a generic library file name to a specific versioned file name. AFAIK this is common practice within Debian and many other Linux systems. Docker's COPY
command does not seem to understand symlinks so instead makes two complete copies of the library files. This results in a larger Docker Image size and warnings from later apt-get
commands to the tune of ldconfig: /usr/local/lib/libprotobuf.so.17 is not a symbolic link
.
My specific file presently looks like:
#Compile any tools we cannot install from packages
FROM gcc:7 as builder
USER 0
RUN
apt-get -y update &&
apt-get -y install
clang
libc++-dev
libgflags-dev
libgtest-dev
RUN
# Protocol Buffer & gRPC
# install protobuf first, then grpc
git clone -b $(curl -L https://grpc.io/release)
https://github.com/grpc/grpc /var/local/git/grpc &&
cd /var/local/git/grpc &&
git submodule update --init &&
echo "--- installing protobuf ---" &&
cd third_party/protobuf &&
./autogen.sh && ./configure --enable-shared &&
make -j$(nproc) && make install && make clean && ldconfig &&
echo "--- installing grpc ---" &&
cd /var/local/git/grpc &&
make -j$(nproc) && make install && make clean && ldconfig
FROM debian
LABEL
Description="Basic Debian production environment with a number of libraries configured"
MAINTAINER="Mr Me"
ARG prefix=/usr/local
ARG binPath=$prefix/bin
ARG libPath=$prefix/lib
# Copy over pre-made tools
# Protocol Buffer
COPY --from=builder /usr/local/lib/libproto* $libPath/
# gRPC
COPY --from=builder /usr/local/lib/libaddress_sorting.so.6.0.0 $libPath/
COPY --from=builder /usr/local/lib/libgpr* $libPath/
COPY --from=builder /usr/local/lib/libgrpc* $libPath/
RUN ldconfig
# Install remaining tools using apt-get
RUN apt-get -y update &&
apt-get -y install
libhdf5-dev
libssl1.1
uuid-dev;
As you can see I am trying to add the latest versions of gRPC and Protocol Buffer to a Debian based runtime image.
c++ docker shared-libraries dockerfile docker-multi-stage-build
add a comment |
I have a Dockerfile which is split into a two-stage multi-stage docker build. The first stage generates a basic gcc build environment in which a number of C and C++ library are compiled. The second stage uses the COPY --from=
command to copy the library files from the first stages /usr/local/lib/libproto*
to the current image's.
The problem I am seeing is that the first image contains symlinks from a generic library file name to a specific versioned file name. AFAIK this is common practice within Debian and many other Linux systems. Docker's COPY
command does not seem to understand symlinks so instead makes two complete copies of the library files. This results in a larger Docker Image size and warnings from later apt-get
commands to the tune of ldconfig: /usr/local/lib/libprotobuf.so.17 is not a symbolic link
.
My specific file presently looks like:
#Compile any tools we cannot install from packages
FROM gcc:7 as builder
USER 0
RUN
apt-get -y update &&
apt-get -y install
clang
libc++-dev
libgflags-dev
libgtest-dev
RUN
# Protocol Buffer & gRPC
# install protobuf first, then grpc
git clone -b $(curl -L https://grpc.io/release)
https://github.com/grpc/grpc /var/local/git/grpc &&
cd /var/local/git/grpc &&
git submodule update --init &&
echo "--- installing protobuf ---" &&
cd third_party/protobuf &&
./autogen.sh && ./configure --enable-shared &&
make -j$(nproc) && make install && make clean && ldconfig &&
echo "--- installing grpc ---" &&
cd /var/local/git/grpc &&
make -j$(nproc) && make install && make clean && ldconfig
FROM debian
LABEL
Description="Basic Debian production environment with a number of libraries configured"
MAINTAINER="Mr Me"
ARG prefix=/usr/local
ARG binPath=$prefix/bin
ARG libPath=$prefix/lib
# Copy over pre-made tools
# Protocol Buffer
COPY --from=builder /usr/local/lib/libproto* $libPath/
# gRPC
COPY --from=builder /usr/local/lib/libaddress_sorting.so.6.0.0 $libPath/
COPY --from=builder /usr/local/lib/libgpr* $libPath/
COPY --from=builder /usr/local/lib/libgrpc* $libPath/
RUN ldconfig
# Install remaining tools using apt-get
RUN apt-get -y update &&
apt-get -y install
libhdf5-dev
libssl1.1
uuid-dev;
As you can see I am trying to add the latest versions of gRPC and Protocol Buffer to a Debian based runtime image.
c++ docker shared-libraries dockerfile docker-multi-stage-build
As a workaround you could tar the files, copy the tarball and then untar them in the second container. This would preserve links.
– Jacob Tomlinson
Nov 13 '18 at 15:46
@JacobTomlinson it is a clunky answer but what you are suggesting could be the only answer. Please, could you give it in the form of an answer?
– TafT
Nov 13 '18 at 16:42
It is so clunky I didn't feel like I should XD. I will do now at your request.
– Jacob Tomlinson
Nov 13 '18 at 16:45
add a comment |
I have a Dockerfile which is split into a two-stage multi-stage docker build. The first stage generates a basic gcc build environment in which a number of C and C++ library are compiled. The second stage uses the COPY --from=
command to copy the library files from the first stages /usr/local/lib/libproto*
to the current image's.
The problem I am seeing is that the first image contains symlinks from a generic library file name to a specific versioned file name. AFAIK this is common practice within Debian and many other Linux systems. Docker's COPY
command does not seem to understand symlinks so instead makes two complete copies of the library files. This results in a larger Docker Image size and warnings from later apt-get
commands to the tune of ldconfig: /usr/local/lib/libprotobuf.so.17 is not a symbolic link
.
My specific file presently looks like:
#Compile any tools we cannot install from packages
FROM gcc:7 as builder
USER 0
RUN
apt-get -y update &&
apt-get -y install
clang
libc++-dev
libgflags-dev
libgtest-dev
RUN
# Protocol Buffer & gRPC
# install protobuf first, then grpc
git clone -b $(curl -L https://grpc.io/release)
https://github.com/grpc/grpc /var/local/git/grpc &&
cd /var/local/git/grpc &&
git submodule update --init &&
echo "--- installing protobuf ---" &&
cd third_party/protobuf &&
./autogen.sh && ./configure --enable-shared &&
make -j$(nproc) && make install && make clean && ldconfig &&
echo "--- installing grpc ---" &&
cd /var/local/git/grpc &&
make -j$(nproc) && make install && make clean && ldconfig
FROM debian
LABEL
Description="Basic Debian production environment with a number of libraries configured"
MAINTAINER="Mr Me"
ARG prefix=/usr/local
ARG binPath=$prefix/bin
ARG libPath=$prefix/lib
# Copy over pre-made tools
# Protocol Buffer
COPY --from=builder /usr/local/lib/libproto* $libPath/
# gRPC
COPY --from=builder /usr/local/lib/libaddress_sorting.so.6.0.0 $libPath/
COPY --from=builder /usr/local/lib/libgpr* $libPath/
COPY --from=builder /usr/local/lib/libgrpc* $libPath/
RUN ldconfig
# Install remaining tools using apt-get
RUN apt-get -y update &&
apt-get -y install
libhdf5-dev
libssl1.1
uuid-dev;
As you can see I am trying to add the latest versions of gRPC and Protocol Buffer to a Debian based runtime image.
c++ docker shared-libraries dockerfile docker-multi-stage-build
I have a Dockerfile which is split into a two-stage multi-stage docker build. The first stage generates a basic gcc build environment in which a number of C and C++ library are compiled. The second stage uses the COPY --from=
command to copy the library files from the first stages /usr/local/lib/libproto*
to the current image's.
The problem I am seeing is that the first image contains symlinks from a generic library file name to a specific versioned file name. AFAIK this is common practice within Debian and many other Linux systems. Docker's COPY
command does not seem to understand symlinks so instead makes two complete copies of the library files. This results in a larger Docker Image size and warnings from later apt-get
commands to the tune of ldconfig: /usr/local/lib/libprotobuf.so.17 is not a symbolic link
.
My specific file presently looks like:
#Compile any tools we cannot install from packages
FROM gcc:7 as builder
USER 0
RUN
apt-get -y update &&
apt-get -y install
clang
libc++-dev
libgflags-dev
libgtest-dev
RUN
# Protocol Buffer & gRPC
# install protobuf first, then grpc
git clone -b $(curl -L https://grpc.io/release)
https://github.com/grpc/grpc /var/local/git/grpc &&
cd /var/local/git/grpc &&
git submodule update --init &&
echo "--- installing protobuf ---" &&
cd third_party/protobuf &&
./autogen.sh && ./configure --enable-shared &&
make -j$(nproc) && make install && make clean && ldconfig &&
echo "--- installing grpc ---" &&
cd /var/local/git/grpc &&
make -j$(nproc) && make install && make clean && ldconfig
FROM debian
LABEL
Description="Basic Debian production environment with a number of libraries configured"
MAINTAINER="Mr Me"
ARG prefix=/usr/local
ARG binPath=$prefix/bin
ARG libPath=$prefix/lib
# Copy over pre-made tools
# Protocol Buffer
COPY --from=builder /usr/local/lib/libproto* $libPath/
# gRPC
COPY --from=builder /usr/local/lib/libaddress_sorting.so.6.0.0 $libPath/
COPY --from=builder /usr/local/lib/libgpr* $libPath/
COPY --from=builder /usr/local/lib/libgrpc* $libPath/
RUN ldconfig
# Install remaining tools using apt-get
RUN apt-get -y update &&
apt-get -y install
libhdf5-dev
libssl1.1
uuid-dev;
As you can see I am trying to add the latest versions of gRPC and Protocol Buffer to a Debian based runtime image.
c++ docker shared-libraries dockerfile docker-multi-stage-build
c++ docker shared-libraries dockerfile docker-multi-stage-build
asked Nov 13 '18 at 13:00
TafTTafT
77211025
77211025
As a workaround you could tar the files, copy the tarball and then untar them in the second container. This would preserve links.
– Jacob Tomlinson
Nov 13 '18 at 15:46
@JacobTomlinson it is a clunky answer but what you are suggesting could be the only answer. Please, could you give it in the form of an answer?
– TafT
Nov 13 '18 at 16:42
It is so clunky I didn't feel like I should XD. I will do now at your request.
– Jacob Tomlinson
Nov 13 '18 at 16:45
add a comment |
As a workaround you could tar the files, copy the tarball and then untar them in the second container. This would preserve links.
– Jacob Tomlinson
Nov 13 '18 at 15:46
@JacobTomlinson it is a clunky answer but what you are suggesting could be the only answer. Please, could you give it in the form of an answer?
– TafT
Nov 13 '18 at 16:42
It is so clunky I didn't feel like I should XD. I will do now at your request.
– Jacob Tomlinson
Nov 13 '18 at 16:45
As a workaround you could tar the files, copy the tarball and then untar them in the second container. This would preserve links.
– Jacob Tomlinson
Nov 13 '18 at 15:46
As a workaround you could tar the files, copy the tarball and then untar them in the second container. This would preserve links.
– Jacob Tomlinson
Nov 13 '18 at 15:46
@JacobTomlinson it is a clunky answer but what you are suggesting could be the only answer. Please, could you give it in the form of an answer?
– TafT
Nov 13 '18 at 16:42
@JacobTomlinson it is a clunky answer but what you are suggesting could be the only answer. Please, could you give it in the form of an answer?
– TafT
Nov 13 '18 at 16:42
It is so clunky I didn't feel like I should XD. I will do now at your request.
– Jacob Tomlinson
Nov 13 '18 at 16:45
It is so clunky I didn't feel like I should XD. I will do now at your request.
– Jacob Tomlinson
Nov 13 '18 at 16:45
add a comment |
1 Answer
1
active
oldest
votes
This is more of a workaround than an answer.
You could tar the files, copy the tarball to the second container and then untar them.
Tar maintains symbolic links by default.
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53281570%2fhow-to-copy-library-files-between-stages-of-a-multi-stage-docker-build-while-pre%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
This is more of a workaround than an answer.
You could tar the files, copy the tarball to the second container and then untar them.
Tar maintains symbolic links by default.
add a comment |
This is more of a workaround than an answer.
You could tar the files, copy the tarball to the second container and then untar them.
Tar maintains symbolic links by default.
add a comment |
This is more of a workaround than an answer.
You could tar the files, copy the tarball to the second container and then untar them.
Tar maintains symbolic links by default.
This is more of a workaround than an answer.
You could tar the files, copy the tarball to the second container and then untar them.
Tar maintains symbolic links by default.
answered Nov 13 '18 at 16:48
Jacob TomlinsonJacob Tomlinson
1,60921946
1,60921946
add a comment |
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53281570%2fhow-to-copy-library-files-between-stages-of-a-multi-stage-docker-build-while-pre%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
As a workaround you could tar the files, copy the tarball and then untar them in the second container. This would preserve links.
– Jacob Tomlinson
Nov 13 '18 at 15:46
@JacobTomlinson it is a clunky answer but what you are suggesting could be the only answer. Please, could you give it in the form of an answer?
– TafT
Nov 13 '18 at 16:42
It is so clunky I didn't feel like I should XD. I will do now at your request.
– Jacob Tomlinson
Nov 13 '18 at 16:45