quality assurance Questions

I am using Azure Databricks with Blob Storage. In my Python Notebook, I wanted to read a raw string using spark.read(). I dont know how to read a string with this methodology, if not this then is there any other way by which I can read a string value directly using spark. Because then I need to write that particular string in a text file using df.write.format("text").save([path]).

Currently I am only able to read contents from a file using spark.read.text([path of file]). So, instead of reading from file I directly wanted to pass a string value.

Similar to QA dept being certified (Eg:ISO/IEX) in service based companies, is there specific certification for QA dept in product based companies?

Also provide details & process on how to get the QA dept certified.

We have a very complex software product that is installed and runs on a Computer or Server.  The Software is managed by a Management Console in the Cloud. 

For years all of our End to End testing was manual.  But this was very resource intensive and time comsuming.  So a couple of years ago we moved to a more agule development approach and built a test automation system to run these End to End tests.  However, we quickly found limitations for example we couldn't have automated tests check what is being displayed in the UI running on the Computer, or check what is being displayed in the Management Console in the Cloud.  So we can never fully replicate in automation what the manual tests were doing.

The result is that we still run, some if not all, of the automated tests manually so check the parts that the automation can't.  So it hasn't really saved us any time or effort.  Also the reliability of the automated tests is an issue due to trhe complexity of the software and it having so many moving parts.  We often run into timing related issues.

Is it possible that some software does not make a suitable candidate for End to End test automation?  If so, what sort of approach should be taken in that situation?  I've been ready about test automation a lot and looked at a number of different tools and systems, but both the tools and the theory all seem to be based around website testing, rather than testing Desktop software.

Many thanks in advance for any thoughts and suggestions. 

In live agile and waterfall project, how can we perform regression testing successfully ?

which test cases we need to execute and how to maintain the regression testing folder ?

should we do full regression testing or partial regression? 

how to prioritise regression testing if time is less? 

what are the best practices to successfully do regression testing in a agile project